2023-10-12T22:15:28.118 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-12T22:15:28.119 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_adking@teuthology --verbose --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi --archive /home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697 -- /home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697/orig.config.yaml 2023-10-12T22:15:28.169 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-12T22:15:28.255 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697 branch: wip-adk2-testing-2023-10-11-1434-pacific description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: adking@redhat.com first_in_suite: false job_id: '7423697' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-adk2-testing-2023-10-11-1434-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 3da7aaee8cb8051dac57683eeb2f18b9da99d2c9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3da7aaee8cb8051dac57683eeb2f18b9da99d2c9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-adk2-testing-2023-10-11-1434-pacific sha1: 3da7aaee8cb8051dac57683eeb2f18b9da99d2c9 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 1447 sha1: 3da7aaee8cb8051dac57683eeb2f18b9da99d2c9 sleep_before_teardown: 0 subset: 1/999 suite: orch:cephadm suite_branch: wip-adk2-testing-2023-10-11-1434-pacific suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3da7aaee8cb8051dac57683eeb2f18b9da99d2c9 targets: smithi131.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDFLy6TIRluuOU2mpRGEf+CPBYDhm02Kpe8KbVhgrHGHoHoTGdSrAkVbX8kXk+HL8vZuj+q8217tuUW8g3RFXjk= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKqNVhDHWHENBC2MC/Z8H7Zf/Gxq9arVln7DmZtvPwArFWpMD0GrRioieAvw9z9E7fsHGA5ZVQC1nz/6g3XO/to= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-12_22:09:36 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-10-12T22:15:28.256 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa; will attempt to use it 2023-10-12T22:15:28.319 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks 2023-10-12T22:15:28.319 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-12T22:15:28.321 INFO:teuthology.task.internal:Checking packages... 2023-10-12T22:15:28.345 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3da7aaee8cb8051dac57683eeb2f18b9da99d2c9' 2023-10-12T22:15:28.345 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-12T22:15:28.346 INFO:teuthology.packaging:ref: None 2023-10-12T22:15:28.346 INFO:teuthology.packaging:tag: None 2023-10-12T22:15:28.347 INFO:teuthology.packaging:branch: wip-adk2-testing-2023-10-11-1434-pacific 2023-10-12T22:15:28.347 INFO:teuthology.packaging:sha1: 3da7aaee8cb8051dac57683eeb2f18b9da99d2c9 2023-10-12T22:15:28.347 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-adk2-testing-2023-10-11-1434-pacific 2023-10-12T22:15:28.487 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-71.g3da7aaee 2023-10-12T22:15:28.488 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-12T22:15:28.494 INFO:teuthology.task.internal:no buildpackages task found 2023-10-12T22:15:28.495 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-12T22:15:28.520 INFO:teuthology.task.internal:Saving configuration 2023-10-12T22:15:28.538 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-12T22:15:28.572 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-12T22:15:28.598 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-12 22:10:28.841347', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDFLy6TIRluuOU2mpRGEf+CPBYDhm02Kpe8KbVhgrHGHoHoTGdSrAkVbX8kXk+HL8vZuj+q8217tuUW8g3RFXjk='} 2023-10-12T22:15:28.619 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-12 22:10:28.842732', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKqNVhDHWHENBC2MC/Z8H7Zf/Gxq9arVln7DmZtvPwArFWpMD0GrRioieAvw9z9E7fsHGA5ZVQC1nz/6g3XO/to='} 2023-10-12T22:15:28.620 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-12T22:15:28.626 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-10-12T22:15:28.626 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-10-12T22:15:28.626 INFO:teuthology.run_tasks:Running task console_log... 2023-10-12T22:15:28.720 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f573020da60>, signals=[15]) 2023-10-12T22:15:28.721 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-12T22:15:28.728 INFO:teuthology.task.internal:Opening connections... 2023-10-12T22:15:28.728 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2023-10-12T22:15:28.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T22:15:28.797 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2023-10-12T22:15:28.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T22:15:28.866 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-12T22:15:28.873 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2023-10-12T22:15:28.897 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2023-10-12T22:15:28.897 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2023-10-12T22:15:28.958 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2023-10-12T22:15:28.959 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="8" 2023-10-12T22:15:28.959 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2023-10-12T22:15:28.959 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2023-10-12T22:15:28.959 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="8" 2023-10-12T22:15:28.960 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el8" 2023-10-12T22:15:28.960 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-12T22:15:28.960 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2023-10-12T22:15:28.960 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-12T22:15:28.960 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2023-10-12T22:15:28.961 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-12T22:15:28.961 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-12T22:15:28.961 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-12T22:15:28.962 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2023-10-12T22:15:28.986 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2023-10-12T22:15:29.007 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2023-10-12T22:15:29.008 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2023-10-12T22:15:29.068 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2023-10-12T22:15:29.068 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8" 2023-10-12T22:15:29.068 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2023-10-12T22:15:29.068 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2023-10-12T22:15:29.069 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8" 2023-10-12T22:15:29.069 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8" 2023-10-12T22:15:29.069 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-12T22:15:29.069 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2023-10-12T22:15:29.070 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-12T22:15:29.070 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2023-10-12T22:15:29.070 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-12T22:15:29.070 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-12T22:15:29.071 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-12T22:15:29.071 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2023-10-12T22:15:29.094 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-12T22:15:29.105 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-12T22:15:29.137 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-12T22:15:29.137 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2023-10-12T22:15:29.140 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2023-10-12T22:15:29.160 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-12T22:15:29.167 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-12T22:15:29.167 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2023-10-12T22:15:29.202 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2023-10-12T22:15:29.229 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-12T22:15:29.287 INFO:teuthology.run_tasks:Running task kernel... 2023-10-12T22:15:29.304 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-12T22:15:29.304 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-12T22:15:29.305 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-12T22:15:29.306 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-12T22:15:29.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T22:15:29.327 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2023-10-12T22:15:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T22:15:29.328 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2023-10-12T22:15:29.388 INFO:teuthology.orchestra.run.smithi195.stdout:4.18.0-489.el8.x86_64 2023-10-12T22:15:29.388 INFO:teuthology.task.kernel:Running kernel on smithi195: 4.18.0-489.el8.x86_64 2023-10-12T22:15:29.389 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2023-10-12T22:15:29.390 INFO:teuthology.orchestra.run.smithi131.stdout:4.18.0-489.el8.x86_64 2023-10-12T22:15:29.390 INFO:teuthology.task.kernel:Running kernel on smithi131: 4.18.0-489.el8.x86_64 2023-10-12T22:15:29.390 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum install -y kernel 2023-10-12T22:15:34.095 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - AppStream 15 MB/s | 33 MB 00:02 2023-10-12T22:15:34.250 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - AppStream 15 MB/s | 33 MB 00:02 2023-10-12T22:15:42.417 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 49 MB 00:02 2023-10-12T22:15:43.403 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 49 MB 00:03 2023-10-12T22:15:49.658 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - Extras 83 kB/s | 18 kB 00:00 2023-10-12T22:15:50.109 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - Extras common packages 25 kB/s | 6.9 kB 00:00 2023-10-12T22:15:50.394 INFO:teuthology.orchestra.run.smithi131.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-10-12T22:15:51.045 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - Extras 28 kB/s | 18 kB 00:00 2023-10-12T22:15:51.481 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - Extras common packages 27 kB/s | 6.9 kB 00:00 2023-10-12T22:15:51.664 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-10-12T22:15:51.771 INFO:teuthology.orchestra.run.smithi195.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-10-12T22:15:53.489 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-10-12T22:15:55.470 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 392 kB/s | 24 kB 00:00 2023-10-12T22:15:57.330 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 413 kB/s | 24 kB 00:00 2023-10-12T22:15:58.970 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-12T22:15:58.971 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-12T22:15:59.238 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:15:59.239 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:15:59.239 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2023-10-12T22:15:59.239 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:15:59.240 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:15:59.240 INFO:teuthology.orchestra.run.smithi131.stdout: kernel x86_64 4.18.0-516.el8 baseos 10 M 2023-10-12T22:15:59.240 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2023-10-12T22:15:59.240 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-core x86_64 4.18.0-516.el8 baseos 43 M 2023-10-12T22:15:59.240 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules x86_64 4.18.0-516.el8 baseos 36 M 2023-10-12T22:15:59.241 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:15:59.241 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:15:59.241 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:15:59.241 INFO:teuthology.orchestra.run.smithi131.stdout:Install 3 Packages 2023-10-12T22:15:59.242 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:15:59.242 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 89 M 2023-10-12T22:15:59.243 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 96 M 2023-10-12T22:15:59.243 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:16:00.471 INFO:teuthology.orchestra.run.smithi131.stdout:(1/3): kernel-4.18.0-516.el8.x86_64.rpm 9.7 MB/s | 10 MB 00:01 2023-10-12T22:16:00.805 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-12T22:16:00.805 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-12T22:16:01.070 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:16:01.070 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:16:01.070 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2023-10-12T22:16:01.071 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:16:01.071 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:16:01.071 INFO:teuthology.orchestra.run.smithi195.stdout: kernel x86_64 4.18.0-516.el8 baseos 10 M 2023-10-12T22:16:01.071 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2023-10-12T22:16:01.072 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core x86_64 4.18.0-516.el8 baseos 43 M 2023-10-12T22:16:01.072 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules x86_64 4.18.0-516.el8 baseos 36 M 2023-10-12T22:16:01.072 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:16:01.072 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:16:01.072 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:16:01.073 INFO:teuthology.orchestra.run.smithi195.stdout:Install 3 Packages 2023-10-12T22:16:01.073 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:16:01.073 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 89 M 2023-10-12T22:16:01.074 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 96 M 2023-10-12T22:16:01.074 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:16:01.605 INFO:teuthology.orchestra.run.smithi131.stdout:(2/3): kernel-modules-4.18.0-516.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2023-10-12T22:16:02.080 INFO:teuthology.orchestra.run.smithi131.stdout:(3/3): kernel-core-4.18.0-516.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2023-10-12T22:16:02.080 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:16:02.081 INFO:teuthology.orchestra.run.smithi131.stdout:Total 31 MB/s | 89 MB 00:02 2023-10-12T22:16:02.754 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:16:03.179 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:16:03.179 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:16:03.548 INFO:teuthology.orchestra.run.smithi195.stdout:(1/3): kernel-4.18.0-516.el8.x86_64.rpm 4.4 MB/s | 10 MB 00:02 2023-10-12T22:16:04.434 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:16:04.453 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:16:05.040 INFO:teuthology.orchestra.run.smithi195.stdout:(2/3): kernel-modules-4.18.0-516.el8.x86_64.rpm 9.4 MB/s | 36 MB 00:03 2023-10-12T22:16:05.612 INFO:teuthology.orchestra.run.smithi195.stdout:(3/3): kernel-core-4.18.0-516.el8.x86_64.rpm 9.9 MB/s | 43 MB 00:04 2023-10-12T22:16:05.612 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:16:05.613 INFO:teuthology.orchestra.run.smithi195.stdout:Total 20 MB/s | 89 MB 00:04 2023-10-12T22:16:06.285 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:16:06.700 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:16:06.700 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:16:07.223 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:16:07.730 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-core-4.18.0-516.el8.x86_64 1/3 2023-10-12T22:16:07.931 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:16:07.961 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:16:10.734 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:16:11.274 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-core-4.18.0-516.el8.x86_64 1/3 2023-10-12T22:16:11.282 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-4.18.0-516.el8.x86_64 1/3 2023-10-12T22:16:11.671 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-modules-4.18.0-516.el8.x86_64 2/3 2023-10-12T22:16:14.826 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-4.18.0-516.el8.x86_64 1/3 2023-10-12T22:16:15.052 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-4.18.0-516.el8.x86_64 2/3 2023-10-12T22:16:15.190 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-4.18.0-516.el8.x86_64 2/3 2023-10-12T22:16:15.328 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-4.18.0-516.el8.x86_64 3/3 2023-10-12T22:16:18.568 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-4.18.0-516.el8.x86_64 2/3 2023-10-12T22:16:18.863 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-4.18.0-516.el8.x86_64 3/3 2023-10-12T22:16:37.122 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-4.18.0-516.el8.x86_64 3/3 2023-10-12T22:16:37.147 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-4.18.0-516.el8.x86_64 3/3 2023-10-12T22:16:37.977 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-4.18.0-516.el8.x86_64 3/3 2023-10-12T22:16:37.977 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-4.18.0-516.el8.x86_64 1/3 2023-10-12T22:16:37.977 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-core-4.18.0-516.el8.x86_64 2/3 2023-10-12T22:16:38.721 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-modules-4.18.0-516.el8.x86_64 3/3 2023-10-12T22:16:38.721 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:16:38.721 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:16:38.722 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-4.18.0-516.el8.x86_64 kernel-core-4.18.0-516.el8.x86_64 2023-10-12T22:16:38.722 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-4.18.0-516.el8.x86_64 2023-10-12T22:16:38.722 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:16:38.722 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:16:38.867 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2023-10-12T22:16:39.129 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-4.18.0-516.el8.x86_64 2023-10-12T22:16:39.129 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-516.el8.x86_64 2023-10-12T22:16:39.130 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-516.el8.x86_64 2023-10-12T22:16:39.130 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-12T22:16:39.130 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-12T22:16:39.130 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-10-12T22:16:39.130 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi131.front.sepia.ceph.com, path=None, version=distro) 2023-10-12T22:16:39.131 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2023-10-12T22:16:39.340 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-4.18.0-516.el8.x86_64 2023-10-12T22:16:39.340 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-516.el8.x86_64 2023-10-12T22:16:39.341 DEBUG:teuthology.orchestra.run.smithi131:> sudo rpm -qi grub2-tools 2023-10-12T22:16:39.375 INFO:teuthology.orchestra.run.smithi131.stdout:Name : grub2-tools 2023-10-12T22:16:39.375 INFO:teuthology.orchestra.run.smithi131.stdout:Epoch : 1 2023-10-12T22:16:39.376 INFO:teuthology.orchestra.run.smithi131.stdout:Version : 2.02 2023-10-12T22:16:39.376 INFO:teuthology.orchestra.run.smithi131.stdout:Release : 129.el8 2023-10-12T22:16:39.376 INFO:teuthology.orchestra.run.smithi131.stdout:Architecture: x86_64 2023-10-12T22:16:39.376 INFO:teuthology.orchestra.run.smithi131.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-12T22:16:39.376 INFO:teuthology.orchestra.run.smithi131.stdout:Group : System Environment/Base 2023-10-12T22:16:39.377 INFO:teuthology.orchestra.run.smithi131.stdout:Size : 9265586 2023-10-12T22:16:39.377 INFO:teuthology.orchestra.run.smithi131.stdout:License : GPLv3+ 2023-10-12T22:16:39.377 INFO:teuthology.orchestra.run.smithi131.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-12T22:16:39.377 INFO:teuthology.orchestra.run.smithi131.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-12T22:16:39.377 INFO:teuthology.orchestra.run.smithi131.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-12T22:16:39.378 INFO:teuthology.orchestra.run.smithi131.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-12T22:16:39.378 INFO:teuthology.orchestra.run.smithi131.stdout:Relocations : (not relocatable) 2023-10-12T22:16:39.378 INFO:teuthology.orchestra.run.smithi131.stdout:Packager : CentOS BuildSystem 2023-10-12T22:16:39.378 INFO:teuthology.orchestra.run.smithi131.stdout:Vendor : CentOS 2023-10-12T22:16:39.378 INFO:teuthology.orchestra.run.smithi131.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-12T22:16:39.379 INFO:teuthology.orchestra.run.smithi131.stdout:Summary : Support tools for GRUB. 2023-10-12T22:16:39.379 INFO:teuthology.orchestra.run.smithi131.stdout:Description : 2023-10-12T22:16:39.379 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:16:39.379 INFO:teuthology.orchestra.run.smithi131.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-12T22:16:39.379 INFO:teuthology.orchestra.run.smithi131.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-12T22:16:39.380 INFO:teuthology.orchestra.run.smithi131.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-12T22:16:39.380 INFO:teuthology.orchestra.run.smithi131.stdout:hardware devices. 2023-10-12T22:16:39.380 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:16:39.380 INFO:teuthology.orchestra.run.smithi131.stdout:This subpackage provides tools for support of all platforms. 2023-10-12T22:16:39.381 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-12T22:16:39.382 INFO:teuthology.task.kernel:Updating grub on smithi131 to boot 4.18.0-516.el8.x86_64 2023-10-12T22:16:39.382 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-12T22:16:40.456 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-4.18.0-516.el8.x86_64 3/3 2023-10-12T22:16:40.481 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-4.18.0-516.el8.x86_64 3/3 2023-10-12T22:16:40.688 INFO:teuthology.orchestra.run.smithi131.stderr:Generating grub configuration file ... 2023-10-12T22:16:40.825 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-4.18.0-516.el8.x86_64 3/3 2023-10-12T22:16:40.826 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-4.18.0-516.el8.x86_64 1/3 2023-10-12T22:16:40.826 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-core-4.18.0-516.el8.x86_64 2/3 2023-10-12T22:16:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-4.18.0-516.el8.x86_64 3/3 2023-10-12T22:16:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:16:41.536 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:16:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-4.18.0-516.el8.x86_64 kernel-core-4.18.0-516.el8.x86_64 2023-10-12T22:16:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-4.18.0-516.el8.x86_64 2023-10-12T22:16:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:16:41.537 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:16:41.680 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2023-10-12T22:16:41.897 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-4.18.0-516.el8.x86_64 2023-10-12T22:16:41.898 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-516.el8.x86_64 2023-10-12T22:16:41.898 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-516.el8.x86_64 2023-10-12T22:16:41.898 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-12T22:16:41.898 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-12T22:16:41.899 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-10-12T22:16:41.899 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2023-10-12T22:16:41.899 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2023-10-12T22:16:42.107 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-4.18.0-516.el8.x86_64 2023-10-12T22:16:42.108 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-516.el8.x86_64 2023-10-12T22:16:42.108 DEBUG:teuthology.orchestra.run.smithi195:> sudo rpm -qi grub2-tools 2023-10-12T22:16:42.142 INFO:teuthology.orchestra.run.smithi195.stdout:Name : grub2-tools 2023-10-12T22:16:42.143 INFO:teuthology.orchestra.run.smithi195.stdout:Epoch : 1 2023-10-12T22:16:42.143 INFO:teuthology.orchestra.run.smithi195.stdout:Version : 2.02 2023-10-12T22:16:42.143 INFO:teuthology.orchestra.run.smithi195.stdout:Release : 129.el8 2023-10-12T22:16:42.143 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: x86_64 2023-10-12T22:16:42.143 INFO:teuthology.orchestra.run.smithi195.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-12T22:16:42.144 INFO:teuthology.orchestra.run.smithi195.stdout:Group : System Environment/Base 2023-10-12T22:16:42.144 INFO:teuthology.orchestra.run.smithi195.stdout:Size : 9265586 2023-10-12T22:16:42.144 INFO:teuthology.orchestra.run.smithi195.stdout:License : GPLv3+ 2023-10-12T22:16:42.144 INFO:teuthology.orchestra.run.smithi195.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-12T22:16:42.144 INFO:teuthology.orchestra.run.smithi195.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-12T22:16:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-12T22:16:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-12T22:16:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:Relocations : (not relocatable) 2023-10-12T22:16:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:Packager : CentOS BuildSystem 2023-10-12T22:16:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:Vendor : CentOS 2023-10-12T22:16:42.146 INFO:teuthology.orchestra.run.smithi195.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-12T22:16:42.146 INFO:teuthology.orchestra.run.smithi195.stdout:Summary : Support tools for GRUB. 2023-10-12T22:16:42.146 INFO:teuthology.orchestra.run.smithi195.stdout:Description : 2023-10-12T22:16:42.147 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:16:42.147 INFO:teuthology.orchestra.run.smithi195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-12T22:16:42.147 INFO:teuthology.orchestra.run.smithi195.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-12T22:16:42.147 INFO:teuthology.orchestra.run.smithi195.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-12T22:16:42.147 INFO:teuthology.orchestra.run.smithi195.stdout:hardware devices. 2023-10-12T22:16:42.148 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:16:42.148 INFO:teuthology.orchestra.run.smithi195.stdout:This subpackage provides tools for support of all platforms. 2023-10-12T22:16:42.149 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-12T22:16:42.149 INFO:teuthology.task.kernel:Updating grub on smithi195 to boot 4.18.0-516.el8.x86_64 2023-10-12T22:16:42.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-12T22:16:43.114 INFO:teuthology.orchestra.run.smithi131.stderr:done 2023-10-12T22:16:43.125 DEBUG:teuthology.orchestra.run.smithi131:> mktemp 2023-10-12T22:16:43.148 INFO:teuthology.orchestra.run.smithi131.stdout:/tmp/tmp.Dg8CEG3yDT 2023-10-12T22:16:43.148 DEBUG:teuthology.orchestra.run.smithi131:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Dg8CEG3yDT 2023-10-12T22:16:43.221 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0666 /tmp/tmp.Dg8CEG3yDT 2023-10-12T22:16:43.362 DEBUG:teuthology.orchestra.remote:smithi131:/tmp/tmp.Dg8CEG3yDT is 6KB 2023-10-12T22:16:43.411 DEBUG:teuthology.orchestra.run.smithi131:> rm -fr /tmp/tmp.Dg8CEG3yDT 2023-10-12T22:16:43.429 DEBUG:teuthology.orchestra.run.smithi131:> sudo /bin/ls /boot/loader/entries || true 2023-10-12T22:16:43.496 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-12T22:16:43.496 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-12T22:16:43.497 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-12T22:16:43.497 INFO:teuthology.orchestra.run.smithi131.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-516.el8.x86_64.conf 2023-10-12T22:16:43.498 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-516.el8.x86_64 2023-10-12T22:16:43.680 DEBUG:teuthology.orchestra.run.smithi131:> sudo shutdown -r now 2023-10-12T22:16:43.818 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2023-10-12T22:16:46.249 INFO:teuthology.orchestra.run.smithi195.stderr:done 2023-10-12T22:16:46.251 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2023-10-12T22:16:46.321 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.HCs2IluK5z 2023-10-12T22:16:46.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HCs2IluK5z 2023-10-12T22:16:46.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.HCs2IluK5z 2023-10-12T22:16:46.543 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.HCs2IluK5z is 6KB 2023-10-12T22:16:46.591 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.HCs2IluK5z 2023-10-12T22:16:46.609 DEBUG:teuthology.orchestra.run.smithi195:> sudo /bin/ls /boot/loader/entries || true 2023-10-12T22:16:46.691 INFO:teuthology.orchestra.run.smithi195.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-12T22:16:46.692 INFO:teuthology.orchestra.run.smithi195.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-12T22:16:46.692 INFO:teuthology.orchestra.run.smithi195.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-12T22:16:46.692 INFO:teuthology.orchestra.run.smithi195.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-516.el8.x86_64.conf 2023-10-12T22:16:46.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-516.el8.x86_64 2023-10-12T22:16:46.842 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2023-10-12T22:17:13.706 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-10-12T22:17:13.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-10-12T22:17:13.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T22:17:16.846 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-10-12T22:17:16.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-10-12T22:17:16.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T22:17:32.026 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.131 2023-10-12T22:17:41.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-10-12T22:17:41.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T22:18:16.886 DEBUG:teuthology.orchestra.remote:timed out 2023-10-12T22:18:25.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-10-12T22:18:25.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T22:18:26.317 DEBUG:teuthology.orchestra.run.smithi195:> true 2023-10-12T22:18:26.644 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-10-12T22:18:26.645 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-516.el8.x86_64"... 2023-10-12T22:18:26.645 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2023-10-12T22:18:26.663 INFO:teuthology.orchestra.run.smithi195.stdout:4.18.0-516.el8.x86_64 2023-10-12T22:18:26.663 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-516.el8.x86_64 vs 4.18.0-516.el8.x86_64 2023-10-12T22:18:26.663 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-12T22:18:26.663 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-12T22:18:27.665 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-10-12T22:18:27.665 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-12T22:18:27.767 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2023-10-12T22:18:27.784 DEBUG:teuthology.parallel:result is None 2023-10-12T22:18:41.038 DEBUG:teuthology.orchestra.remote:timed out 2023-10-12T22:18:53.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-10-12T22:18:53.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T22:18:53.389 DEBUG:teuthology.orchestra.run.smithi131:> true 2023-10-12T22:18:53.697 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-10-12T22:18:53.698 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-516.el8.x86_64"... 2023-10-12T22:18:53.698 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2023-10-12T22:18:53.718 INFO:teuthology.orchestra.run.smithi131.stdout:4.18.0-516.el8.x86_64 2023-10-12T22:18:53.719 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-516.el8.x86_64 vs 4.18.0-516.el8.x86_64 2023-10-12T22:18:53.719 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-12T22:18:53.719 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-12T22:18:54.720 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-10-12T22:18:54.720 DEBUG:teuthology.orchestra.run.smithi131:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-12T22:18:54.793 INFO:teuthology.orchestra.run.smithi131.stdout:ttyS1 2023-10-12T22:18:54.809 DEBUG:teuthology.parallel:result is None 2023-10-12T22:18:54.809 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-12T22:18:54.816 INFO:teuthology.task.internal:Creating test directory... 2023-10-12T22:18:54.817 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-12T22:18:54.820 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-12T22:18:54.845 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-12T22:18:54.857 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-12T22:18:54.899 INFO:teuthology.task.internal:Creating archive directory... 2023-10-12T22:18:54.900 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-12T22:18:54.902 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-12T22:18:54.936 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-12T22:18:54.943 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-12T22:18:54.943 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-12T22:18:54.973 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-12T22:18:55.011 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-12T22:18:55.013 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-12T22:18:55.022 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-12T22:18:55.024 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-12T22:18:55.026 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-12T22:18:55.033 INFO:teuthology.task.internal:Configuring sudo... 2023-10-12T22:18:55.033 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-12T22:18:55.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-12T22:18:55.097 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-12T22:18:55.105 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-12T22:18:55.106 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-12T22:18:55.137 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-12T22:18:55.157 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-12T22:18:55.241 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-12T22:18:55.292 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-12T22:18:55.377 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-12T22:18:55.421 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-10-12T22:18:55.421 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-12T22:18:55.489 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-12T22:18:55.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-12T22:18:55.596 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-12T22:18:55.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-12T22:18:55.728 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-12T22:18:55.729 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-12T22:18:55.797 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2023-10-12T22:18:55.800 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2023-10-12T22:18:55.868 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-12T22:18:55.906 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-12T22:18:56.252 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-12T22:18:56.260 INFO:teuthology.task.internal:Starting timer... 2023-10-12T22:18:56.260 INFO:teuthology.run_tasks:Running task pcp... 2023-10-12T22:18:56.303 INFO:teuthology.run_tasks:Running task selinux... 2023-10-12T22:18:56.352 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-12T22:18:56.353 DEBUG:teuthology.orchestra.run.smithi131:> sudo service auditd rotate 2023-10-12T22:18:56.444 INFO:teuthology.orchestra.run.smithi131.stdout:Rotating logs: 2023-10-12T22:18:56.446 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2023-10-12T22:18:56.522 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2023-10-12T22:18:56.524 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-12T22:18:56.524 DEBUG:teuthology.orchestra.run.smithi131:> /usr/sbin/getenforce 2023-10-12T22:18:56.562 INFO:teuthology.orchestra.run.smithi131.stdout:Permissive 2023-10-12T22:18:56.562 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2023-10-12T22:18:56.586 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2023-10-12T22:18:56.587 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi131.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2023-10-12T22:18:56.587 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-12T22:18:56.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T22:18:56.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-12T22:18:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T22:18:56.660 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-12T22:18:56.660 DEBUG:teuthology.orchestra.run.smithi131:> sudo /usr/sbin/setenforce permissive 2023-10-12T22:18:56.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2023-10-12T22:18:56.746 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-12T22:18:56.756 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-12T22:18:56.763 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-12T22:18:56.879 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-12T22:18:56.898 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-12T22:18:56.900 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi131.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-12T22:26:23.713 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2023-10-12T22:26:23.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-10-12T22:26:23.716 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T22:26:23.788 DEBUG:teuthology.orchestra.run.smithi131:> true 2023-10-12T22:26:23.856 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-10-12T22:26:23.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-10-12T22:26:23.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-12T22:26:23.929 DEBUG:teuthology.orchestra.run.smithi195:> true 2023-10-12T22:26:24.003 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-10-12T22:26:24.003 INFO:teuthology.run_tasks:Running task clock... 2023-10-12T22:26:24.013 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-12T22:26:24.014 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-12T22:26:24.015 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-12T22:26:24.017 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-12T22:26:24.017 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-12T22:26:24.046 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-12T22:26:24.060 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-12T22:26:24.079 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-12T22:26:24.094 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-12T22:26:24.118 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: ntpd: command not found 2023-10-12T22:26:24.129 INFO:teuthology.orchestra.run.smithi131.stdout:506 Cannot talk to daemon 2023-10-12T22:26:24.143 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-12T22:26:24.157 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-12T22:26:24.173 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2023-10-12T22:26:24.186 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2023-10-12T22:26:24.201 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-12T22:26:24.214 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-12T22:26:24.223 INFO:teuthology.orchestra.run.smithi131.stderr:bash: ntpq: command not found 2023-10-12T22:26:24.226 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-12T22:26:24.227 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2023-10-12T22:26:24.227 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T22:26:24.227 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T22:26:24.227 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T22:26:24.227 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T22:26:24.294 INFO:teuthology.orchestra.run.smithi195.stderr:bash: ntpq: command not found 2023-10-12T22:26:24.298 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-12T22:26:24.298 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2023-10-12T22:26:24.298 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T22:26:24.299 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T22:26:24.299 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T22:26:24.299 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T22:26:24.299 INFO:teuthology.run_tasks:Running task pexec... 2023-10-12T22:26:24.309 INFO:teuthology.task.pexec:Executing custom commands... 2023-10-12T22:26:24.310 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi131.front.sepia.ceph.com 2023-10-12T22:26:24.310 DEBUG:teuthology.orchestra.run.smithi131:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-12T22:26:24.311 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi195.front.sepia.ceph.com 2023-10-12T22:26:24.311 DEBUG:teuthology.orchestra.run.smithi195:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-12T22:26:25.192 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:37 ago on Thu 12 Oct 2023 10:24:48 PM UTC. 2023-10-12T22:26:25.223 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:37 ago on Thu 12 Oct 2023 10:24:48 PM UTC. 2023-10-12T22:26:27.210 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:26:27.211 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:26:27.211 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2023-10-12T22:26:27.212 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:26:27.212 INFO:teuthology.orchestra.run.smithi131.stdout:Resetting modules: 2023-10-12T22:26:27.212 INFO:teuthology.orchestra.run.smithi131.stdout: container-tools 2023-10-12T22:26:27.213 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:26:27.213 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:26:27.213 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:26:27.214 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:26:27.282 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:26:27.285 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:26:27.285 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2023-10-12T22:26:27.285 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:26:27.286 INFO:teuthology.orchestra.run.smithi195.stdout:Resetting modules: 2023-10-12T22:26:27.286 INFO:teuthology.orchestra.run.smithi195.stdout: container-tools 2023-10-12T22:26:27.286 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:26:27.286 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:26:27.286 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:26:27.287 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:26:27.461 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:26:27.532 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:26:28.411 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:40 ago on Thu 12 Oct 2023 10:24:48 PM UTC. 2023-10-12T22:26:28.487 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:40 ago on Thu 12 Oct 2023 10:24:48 PM UTC. 2023-10-12T22:26:30.427 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:26:30.591 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:26:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2023-10-12T22:26:30.591 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:26:30.592 INFO:teuthology.orchestra.run.smithi131.stdout:Installing group/module packages: 2023-10-12T22:26:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: aardvark-dns x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 1.0 M 2023-10-12T22:26:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: buildah x86_64 1:1.31.3-1.module_el8+664+4072b3ae appstream 8.8 M 2023-10-12T22:26:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: cockpit-podman noarch 75-1.module_el8+664+4072b3ae appstream 739 k 2023-10-12T22:26:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: crun x86_64 1.8.7-1.module_el8+661+d1afb926 appstream 239 k 2023-10-12T22:26:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: netavark x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 3.7 M 2023-10-12T22:26:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: python3-podman noarch 4.6.0-1.module_el8+582+a2558ae2 appstream 177 k 2023-10-12T22:26:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: skopeo x86_64 2:1.13.3-1.module_el8+664+4072b3ae appstream 8.1 M 2023-10-12T22:26:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: toolbox x86_64 0.0.99.4-5.module_el8+649+e2deac1d appstream 2.5 M 2023-10-12T22:26:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: udica noarch 0.2.6-20.module_el8+487+8e42a277 appstream 53 k 2023-10-12T22:26:30.594 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2023-10-12T22:26:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-10-12T22:26:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-10-12T22:26:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-10-12T22:26:30.595 INFO:teuthology.orchestra.run.smithi131.stdout:Installing weak dependencies: 2023-10-12T22:26:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-10-12T22:26:30.595 INFO:teuthology.orchestra.run.smithi131.stdout:Installing module profiles: 2023-10-12T22:26:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: container-tools/common 2023-10-12T22:26:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:26:30.596 INFO:teuthology.orchestra.run.smithi131.stdout:Enabling module streams: 2023-10-12T22:26:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: container-tools rhel8 2023-10-12T22:26:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:26:30.596 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:26:30.596 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:26:30.597 INFO:teuthology.orchestra.run.smithi131.stdout:Install 13 Packages 2023-10-12T22:26:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:26:30.597 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:26:30.598 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 26 M 2023-10-12T22:26:30.598 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 87 M 2023-10-12T22:26:30.598 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:26:30.752 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:26:30.753 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2023-10-12T22:26:30.753 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:26:30.753 INFO:teuthology.orchestra.run.smithi195.stdout:Installing group/module packages: 2023-10-12T22:26:30.753 INFO:teuthology.orchestra.run.smithi195.stdout: aardvark-dns x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 1.0 M 2023-10-12T22:26:30.754 INFO:teuthology.orchestra.run.smithi195.stdout: buildah x86_64 1:1.31.3-1.module_el8+664+4072b3ae appstream 8.8 M 2023-10-12T22:26:30.754 INFO:teuthology.orchestra.run.smithi195.stdout: cockpit-podman noarch 75-1.module_el8+664+4072b3ae appstream 739 k 2023-10-12T22:26:30.754 INFO:teuthology.orchestra.run.smithi195.stdout: crun x86_64 1.8.7-1.module_el8+661+d1afb926 appstream 239 k 2023-10-12T22:26:30.754 INFO:teuthology.orchestra.run.smithi195.stdout: netavark x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 3.7 M 2023-10-12T22:26:30.754 INFO:teuthology.orchestra.run.smithi195.stdout: python3-podman noarch 4.6.0-1.module_el8+582+a2558ae2 appstream 177 k 2023-10-12T22:26:30.755 INFO:teuthology.orchestra.run.smithi195.stdout: skopeo x86_64 2:1.13.3-1.module_el8+664+4072b3ae appstream 8.1 M 2023-10-12T22:26:30.755 INFO:teuthology.orchestra.run.smithi195.stdout: toolbox x86_64 0.0.99.4-5.module_el8+649+e2deac1d appstream 2.5 M 2023-10-12T22:26:30.755 INFO:teuthology.orchestra.run.smithi195.stdout: udica noarch 0.2.6-20.module_el8+487+8e42a277 appstream 53 k 2023-10-12T22:26:30.755 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2023-10-12T22:26:30.755 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-10-12T22:26:30.756 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-10-12T22:26:30.756 INFO:teuthology.orchestra.run.smithi195.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-10-12T22:26:30.756 INFO:teuthology.orchestra.run.smithi195.stdout:Installing weak dependencies: 2023-10-12T22:26:30.756 INFO:teuthology.orchestra.run.smithi195.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-10-12T22:26:30.756 INFO:teuthology.orchestra.run.smithi195.stdout:Installing module profiles: 2023-10-12T22:26:30.757 INFO:teuthology.orchestra.run.smithi195.stdout: container-tools/common 2023-10-12T22:26:30.757 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:26:30.757 INFO:teuthology.orchestra.run.smithi195.stdout:Enabling module streams: 2023-10-12T22:26:30.757 INFO:teuthology.orchestra.run.smithi195.stdout: container-tools rhel8 2023-10-12T22:26:30.757 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:26:30.757 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:26:30.758 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:26:30.758 INFO:teuthology.orchestra.run.smithi195.stdout:Install 13 Packages 2023-10-12T22:26:30.758 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:26:30.759 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 26 M 2023-10-12T22:26:30.759 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 87 M 2023-10-12T22:26:30.759 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:26:31.182 INFO:teuthology.orchestra.run.smithi131.stdout:(1/13): cockpit-podman-75-1.module_el8+664+4072 1.3 MB/s | 739 kB 00:00 2023-10-12T22:26:31.283 INFO:teuthology.orchestra.run.smithi131.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 378 kB/s | 38 kB 00:00 2023-10-12T22:26:31.316 INFO:teuthology.orchestra.run.smithi131.stdout:(3/13): aardvark-dns-1.7.0-1.module_el8+487+8e4 1.5 MB/s | 1.0 MB 00:00 2023-10-12T22:26:31.417 INFO:teuthology.orchestra.run.smithi131.stdout:(4/13): crun-1.8.7-1.module_el8+661+d1afb926.x8 1.7 MB/s | 239 kB 00:00 2023-10-12T22:26:31.515 INFO:teuthology.orchestra.run.smithi195.stdout:(1/13): cockpit-podman-75-1.module_el8+664+4072 1.3 MB/s | 739 kB 00:00 2023-10-12T22:26:31.525 INFO:teuthology.orchestra.run.smithi131.stdout:(5/13): python3-podman-4.6.0-1.module_el8+582+a 1.6 MB/s | 177 kB 00:00 2023-10-12T22:26:31.615 INFO:teuthology.orchestra.run.smithi195.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 379 kB/s | 38 kB 00:00 2023-10-12T22:26:31.617 INFO:teuthology.orchestra.run.smithi131.stdout:(6/13): netavark-1.7.0-1.module_el8+487+8e42a27 12 MB/s | 3.7 MB 00:00 2023-10-12T22:26:31.643 INFO:teuthology.orchestra.run.smithi131.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 212 kB/s | 25 kB 00:00 2023-10-12T22:26:31.649 INFO:teuthology.orchestra.run.smithi195.stdout:(3/13): aardvark-dns-1.7.0-1.module_el8+487+8e4 1.4 MB/s | 1.0 MB 00:00 2023-10-12T22:26:31.716 INFO:teuthology.orchestra.run.smithi195.stdout:(4/13): crun-1.8.7-1.module_el8+661+d1afb926.x8 2.3 MB/s | 239 kB 00:00 2023-10-12T22:26:31.760 INFO:teuthology.orchestra.run.smithi131.stdout:(8/13): buildah-1.31.3-1.module_el8+664+4072b3a 7.8 MB/s | 8.8 MB 00:01 2023-10-12T22:26:31.785 INFO:teuthology.orchestra.run.smithi131.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 561 kB/s | 94 kB 00:00 2023-10-12T22:26:31.816 INFO:teuthology.orchestra.run.smithi195.stdout:(5/13): python3-podman-4.6.0-1.module_el8+582+a 1.7 MB/s | 177 kB 00:00 2023-10-12T22:26:31.927 INFO:teuthology.orchestra.run.smithi131.stdout:(10/13): toolbox-0.0.99.4-5.module_el8+649+e2de 15 MB/s | 2.5 MB 00:00 2023-10-12T22:26:31.944 INFO:teuthology.orchestra.run.smithi131.stdout:(11/13): udica-0.2.6-20.module_el8+487+8e42a277 332 kB/s | 53 kB 00:00 2023-10-12T22:26:31.950 INFO:teuthology.orchestra.run.smithi195.stdout:(6/13): netavark-1.7.0-1.module_el8+487+8e42a27 12 MB/s | 3.7 MB 00:00 2023-10-12T22:26:31.975 INFO:teuthology.orchestra.run.smithi195.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 156 kB/s | 25 kB 00:00 2023-10-12T22:26:32.045 INFO:teuthology.orchestra.run.smithi131.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 350 kB/s | 41 kB 00:00 2023-10-12T22:26:32.109 INFO:teuthology.orchestra.run.smithi195.stdout:(8/13): buildah-1.31.3-1.module_el8+664+4072b3a 7.6 MB/s | 8.8 MB 00:01 2023-10-12T22:26:32.134 INFO:teuthology.orchestra.run.smithi195.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 510 kB/s | 94 kB 00:00 2023-10-12T22:26:32.211 INFO:teuthology.orchestra.run.smithi131.stdout:(13/13): skopeo-1.13.3-1.module_el8+664+4072b3a 14 MB/s | 8.1 MB 00:00 2023-10-12T22:26:32.212 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:26:32.212 INFO:teuthology.orchestra.run.smithi131.stdout:Total 16 MB/s | 26 MB 00:01 2023-10-12T22:26:32.243 INFO:teuthology.orchestra.run.smithi195.stdout:(10/13): udica-0.2.6-20.module_el8+487+8e42a277 486 kB/s | 53 kB 00:00 2023-10-12T22:26:32.361 INFO:teuthology.orchestra.run.smithi195.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 348 kB/s | 41 kB 00:00 2023-10-12T22:26:32.474 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:26:32.495 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:26:32.496 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:26:32.535 INFO:teuthology.orchestra.run.smithi195.stdout:(12/13): toolbox-0.0.99.4-5.module_el8+649+e2de 5.9 MB/s | 2.5 MB 00:00 2023-10-12T22:26:32.727 INFO:teuthology.orchestra.run.smithi195.stdout:(13/13): skopeo-1.13.3-1.module_el8+664+4072b3a 11 MB/s | 8.1 MB 00:00 2023-10-12T22:26:32.727 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:26:32.728 INFO:teuthology.orchestra.run.smithi195.stdout:Total 13 MB/s | 26 MB 00:01 2023-10-12T22:26:32.768 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:26:32.772 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:26:32.991 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:26:33.012 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:26:33.012 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:26:33.264 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:26:33.268 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:26:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:26:33.660 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-10-12T22:26:33.863 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-10-12T22:26:34.005 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:26:34.104 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-10-12T22:26:34.251 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-10-12T22:26:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-10-12T22:26:34.446 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-10-12T22:26:34.693 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-10-12T22:26:34.952 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 5/13 2023-10-12T22:26:34.957 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-10-12T22:26:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 5/13 2023-10-12T22:26:35.887 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-10-12T22:26:36.140 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 7/13 2023-10-12T22:26:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 8/13 2023-10-12T22:26:36.445 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-10-12T22:26:36.642 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 9/13 2023-10-12T22:26:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 7/13 2023-10-12T22:26:36.946 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 8/13 2023-10-12T22:26:37.056 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : udica-0.2.6-20.module_el8+487+8e42a277.noarch 10/13 2023-10-12T22:26:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 9/13 2023-10-12T22:26:37.645 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : udica-0.2.6-20.module_el8+487+8e42a277.noarch 10/13 2023-10-12T22:26:37.965 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-10-12T22:26:38.197 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 12/13 2023-10-12T22:26:38.403 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-10-12T22:26:38.532 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-10-12T22:26:38.762 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 12/13 2023-10-12T22:26:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-10-12T22:26:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 1/13 2023-10-12T22:26:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2/13 2023-10-12T22:26:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 3/13 2023-10-12T22:26:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-10-12T22:26:38.840 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 5/13 2023-10-12T22:26:38.840 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-10-12T22:26:38.840 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 7/13 2023-10-12T22:26:38.840 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-10-12T22:26:38.840 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-10-12T22:26:38.841 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 10/13 2023-10-12T22:26:38.841 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-10-12T22:26:38.841 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : udica-0.2.6-20.module_el8+487+8e42a277.noarch 12/13 2023-10-12T22:26:38.986 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-10-12T22:26:39.412 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-10-12T22:26:39.413 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 1/13 2023-10-12T22:26:39.413 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2/13 2023-10-12T22:26:39.413 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 3/13 2023-10-12T22:26:39.413 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-10-12T22:26:39.414 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 5/13 2023-10-12T22:26:39.414 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-10-12T22:26:39.414 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 7/13 2023-10-12T22:26:39.414 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-10-12T22:26:39.414 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-10-12T22:26:39.415 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 10/13 2023-10-12T22:26:39.415 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-10-12T22:26:39.415 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : udica-0.2.6-20.module_el8+487+8e42a277.noarch 12/13 2023-10-12T22:26:39.583 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-10-12T22:26:39.584 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:26:39.584 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:26:39.584 INFO:teuthology.orchestra.run.smithi131.stdout: aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-10-12T22:26:39.584 INFO:teuthology.orchestra.run.smithi131.stdout: buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2023-10-12T22:26:39.585 INFO:teuthology.orchestra.run.smithi131.stdout: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 2023-10-12T22:26:39.585 INFO:teuthology.orchestra.run.smithi131.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-10-12T22:26:39.585 INFO:teuthology.orchestra.run.smithi131.stdout: crun-1.8.7-1.module_el8+661+d1afb926.x86_64 2023-10-12T22:26:39.585 INFO:teuthology.orchestra.run.smithi131.stdout: netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-10-12T22:26:39.585 INFO:teuthology.orchestra.run.smithi131.stdout: python3-podman-4.6.0-1.module_el8+582+a2558ae2.noarch 2023-10-12T22:26:39.586 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-10-12T22:26:39.586 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-10-12T22:26:39.586 INFO:teuthology.orchestra.run.smithi131.stdout: skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 2023-10-12T22:26:39.586 INFO:teuthology.orchestra.run.smithi131.stdout: toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 2023-10-12T22:26:39.586 INFO:teuthology.orchestra.run.smithi131.stdout: udica-0.2.6-20.module_el8+487+8e42a277.noarch 2023-10-12T22:26:39.587 INFO:teuthology.orchestra.run.smithi131.stdout: yajl-2.1.0-12.el8.x86_64 2023-10-12T22:26:39.587 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:26:39.587 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:26:39.882 DEBUG:teuthology.parallel:result is None 2023-10-12T22:26:40.579 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-10-12T22:26:40.580 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:26:40.580 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:26:40.580 INFO:teuthology.orchestra.run.smithi195.stdout: aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-10-12T22:26:40.580 INFO:teuthology.orchestra.run.smithi195.stdout: buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2023-10-12T22:26:40.581 INFO:teuthology.orchestra.run.smithi195.stdout: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 2023-10-12T22:26:40.581 INFO:teuthology.orchestra.run.smithi195.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-10-12T22:26:40.581 INFO:teuthology.orchestra.run.smithi195.stdout: crun-1.8.7-1.module_el8+661+d1afb926.x86_64 2023-10-12T22:26:40.581 INFO:teuthology.orchestra.run.smithi195.stdout: netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-10-12T22:26:40.581 INFO:teuthology.orchestra.run.smithi195.stdout: python3-podman-4.6.0-1.module_el8+582+a2558ae2.noarch 2023-10-12T22:26:40.582 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-10-12T22:26:40.582 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-10-12T22:26:40.582 INFO:teuthology.orchestra.run.smithi195.stdout: skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 2023-10-12T22:26:40.582 INFO:teuthology.orchestra.run.smithi195.stdout: toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 2023-10-12T22:26:40.583 INFO:teuthology.orchestra.run.smithi195.stdout: udica-0.2.6-20.module_el8+487+8e42a277.noarch 2023-10-12T22:26:40.583 INFO:teuthology.orchestra.run.smithi195.stdout: yajl-2.1.0-12.el8.x86_64 2023-10-12T22:26:40.583 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:26:40.583 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:26:40.892 DEBUG:teuthology.parallel:result is None 2023-10-12T22:26:40.892 INFO:teuthology.run_tasks:Running task pexec... 2023-10-12T22:26:40.900 INFO:teuthology.task.pexec:Executing custom commands... 2023-10-12T22:26:40.901 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi131.front.sepia.ceph.com 2023-10-12T22:26:40.901 DEBUG:teuthology.orchestra.run.smithi131:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-12T22:26:40.902 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi195.front.sepia.ceph.com 2023-10-12T22:26:40.902 DEBUG:teuthology.orchestra.run.smithi195:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-12T22:26:41.081 DEBUG:teuthology.parallel:result is None 2023-10-12T22:26:41.086 DEBUG:teuthology.parallel:result is None 2023-10-12T22:26:41.086 INFO:teuthology.run_tasks:Running task install... 2023-10-12T22:26:41.095 DEBUG:teuthology.task.install:project ceph 2023-10-12T22:26:41.095 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3da7aaee8cb8051dac57683eeb2f18b9da99d2c9'}} 2023-10-12T22:26:41.096 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '3da7aaee8cb8051dac57683eeb2f18b9da99d2c9'} 2023-10-12T22:26:41.096 INFO:teuthology.task.install:Using flavor: default 2023-10-12T22:26:41.135 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-12T22:26:41.136 INFO:teuthology.task.install:extra packages: [] 2023-10-12T22:26:41.136 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3da7aaee8cb8051dac57683eeb2f18b9da99d2c9', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-10-12T22:26:41.137 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-10-12T22:26:41.137 INFO:teuthology.packaging:ref: None 2023-10-12T22:26:41.137 INFO:teuthology.packaging:tag: v16.2.4 2023-10-12T22:26:41.138 INFO:teuthology.packaging:branch: None 2023-10-12T22:26:41.138 INFO:teuthology.packaging:sha1: 3da7aaee8cb8051dac57683eeb2f18b9da99d2c9 2023-10-12T22:26:41.331 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-10-12T22:26:41.332 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-10-12T22:26:41.334 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3da7aaee8cb8051dac57683eeb2f18b9da99d2c9', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-10-12T22:26:41.335 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-10-12T22:26:41.335 INFO:teuthology.packaging:ref: None 2023-10-12T22:26:41.335 INFO:teuthology.packaging:tag: v16.2.4 2023-10-12T22:26:41.336 INFO:teuthology.packaging:branch: None 2023-10-12T22:26:41.336 INFO:teuthology.packaging:sha1: 3da7aaee8cb8051dac57683eeb2f18b9da99d2c9 2023-10-12T22:26:41.542 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-10-12T22:26:41.543 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-10-12T22:26:41.688 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-10-12T22:26:41.689 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-10-12T22:26:41.755 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-12T22:26:41.755 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-10-12T22:26:41.755 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-12T22:26:41.766 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-10-12T22:26:41.766 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-10-12T22:26:41.790 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-12T22:26:41.790 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-10-12T22:26:41.791 INFO:teuthology.packaging:ref: None 2023-10-12T22:26:41.791 INFO:teuthology.packaging:tag: v16.2.4 2023-10-12T22:26:41.791 INFO:teuthology.packaging:branch: None 2023-10-12T22:26:41.791 INFO:teuthology.packaging:sha1: 3da7aaee8cb8051dac57683eeb2f18b9da99d2c9 2023-10-12T22:26:41.792 DEBUG:teuthology.orchestra.run.smithi131:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-12T22:26:41.794 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-12T22:26:41.795 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-12T22:26:41.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-12T22:26:41.826 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-12T22:26:41.827 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-10-12T22:26:41.827 INFO:teuthology.packaging:ref: None 2023-10-12T22:26:41.827 INFO:teuthology.packaging:tag: v16.2.4 2023-10-12T22:26:41.828 INFO:teuthology.packaging:branch: None 2023-10-12T22:26:41.828 INFO:teuthology.packaging:sha1: 3da7aaee8cb8051dac57683eeb2f18b9da99d2c9 2023-10-12T22:26:41.828 DEBUG:teuthology.orchestra.run.smithi195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-12T22:26:41.862 DEBUG:teuthology.orchestra.run.smithi131:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-12T22:26:41.898 DEBUG:teuthology.orchestra.run.smithi195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-12T22:26:41.943 DEBUG:teuthology.orchestra.run.smithi131:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-12T22:26:41.977 DEBUG:teuthology.orchestra.run.smithi195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-12T22:26:42.012 INFO:teuthology.orchestra.run.smithi131.stdout:check_obsoletes = 1 2023-10-12T22:26:42.013 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum clean all 2023-10-12T22:26:42.045 INFO:teuthology.orchestra.run.smithi195.stdout:check_obsoletes = 1 2023-10-12T22:26:42.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum clean all 2023-10-12T22:26:42.295 INFO:teuthology.orchestra.run.smithi131.stdout:76 files removed 2023-10-12T22:26:42.313 INFO:teuthology.orchestra.run.smithi195.stdout:76 files removed 2023-10-12T22:26:42.325 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-radosgw 2023-10-12T22:26:42.341 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw 2023-10-12T22:26:42.813 INFO:teuthology.orchestra.run.smithi131.stdout:ceph packages for x86_64 938 kB/s | 74 kB 00:00 2023-10-12T22:26:42.836 INFO:teuthology.orchestra.run.smithi195.stdout:ceph packages for x86_64 882 kB/s | 74 kB 00:00 2023-10-12T22:26:42.947 INFO:teuthology.orchestra.run.smithi131.stdout:ceph noarch packages 158 kB/s | 15 kB 00:00 2023-10-12T22:26:42.962 INFO:teuthology.orchestra.run.smithi195.stdout:ceph noarch packages 173 kB/s | 15 kB 00:00 2023-10-12T22:26:43.064 INFO:teuthology.orchestra.run.smithi131.stdout:ceph source packages 20 kB/s | 1.7 kB 00:00 2023-10-12T22:26:43.087 INFO:teuthology.orchestra.run.smithi195.stdout:ceph source packages 18 kB/s | 1.7 kB 00:00 2023-10-12T22:26:45.156 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - AppStream 16 MB/s | 33 MB 00:02 2023-10-12T22:26:45.171 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - AppStream 16 MB/s | 33 MB 00:02 2023-10-12T22:26:52.758 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - BaseOS 32 MB/s | 49 MB 00:01 2023-10-12T22:26:54.564 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 49 MB 00:03 2023-10-12T22:27:00.107 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2023-10-12T22:27:00.525 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - Extras common packages 29 kB/s | 6.9 kB 00:00 2023-10-12T22:27:01.268 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 8 - PowerTools 11 MB/s | 6.5 MB 00:00 2023-10-12T22:27:01.964 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - Extras 47 kB/s | 18 kB 00:00 2023-10-12T22:27:02.423 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - Extras common packages 25 kB/s | 6.9 kB 00:00 2023-10-12T22:27:02.794 INFO:teuthology.orchestra.run.smithi131.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-10-12T22:27:03.878 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-10-12T22:27:04.474 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - PowerTools 3.4 MB/s | 6.5 MB 00:01 2023-10-12T22:27:06.134 INFO:teuthology.orchestra.run.smithi195.stdout:Copr repo for python3-asyncssh owned by ceph 9.3 kB/s | 3.5 kB 00:00 2023-10-12T22:27:07.437 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-10-12T22:27:07.704 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 404 kB/s | 24 kB 00:00 2023-10-12T22:27:11.344 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 224 kB/s | 24 kB 00:00 2023-10-12T22:27:12.110 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:27:12.113 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:27:12.113 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2023-10-12T22:27:12.113 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:27:12.114 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:27:12.114 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-10-12T22:27:12.114 INFO:teuthology.orchestra.run.smithi131.stdout:Upgrading: 2023-10-12T22:27:12.114 INFO:teuthology.orchestra.run.smithi131.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-10-12T22:27:12.115 INFO:teuthology.orchestra.run.smithi131.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-10-12T22:27:12.115 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2023-10-12T22:27:12.115 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-10-12T22:27:12.115 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-10-12T22:27:12.115 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-10-12T22:27:12.116 INFO:teuthology.orchestra.run.smithi131.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-10-12T22:27:12.116 INFO:teuthology.orchestra.run.smithi131.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-10-12T22:27:12.116 INFO:teuthology.orchestra.run.smithi131.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-12T22:27:12.116 INFO:teuthology.orchestra.run.smithi131.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-10-12T22:27:12.117 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-10-12T22:27:12.117 INFO:teuthology.orchestra.run.smithi131.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-12T22:27:12.117 INFO:teuthology.orchestra.run.smithi131.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-10-12T22:27:12.117 INFO:teuthology.orchestra.run.smithi131.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-12T22:27:12.117 INFO:teuthology.orchestra.run.smithi131.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-10-12T22:27:12.118 INFO:teuthology.orchestra.run.smithi131.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-12T22:27:12.118 INFO:teuthology.orchestra.run.smithi131.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-10-12T22:27:12.118 INFO:teuthology.orchestra.run.smithi131.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-12T22:27:12.119 INFO:teuthology.orchestra.run.smithi131.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-12T22:27:12.119 INFO:teuthology.orchestra.run.smithi131.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-10-12T22:27:12.119 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-10-12T22:27:12.119 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-10-12T22:27:12.119 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-10-12T22:27:12.120 INFO:teuthology.orchestra.run.smithi131.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-12T22:27:12.120 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-10-12T22:27:12.120 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-10-12T22:27:12.120 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-10-12T22:27:12.120 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:27:12.121 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:27:12.121 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:27:12.121 INFO:teuthology.orchestra.run.smithi131.stdout:Install 25 Packages 2023-10-12T22:27:12.121 INFO:teuthology.orchestra.run.smithi131.stdout:Upgrade 2 Packages 2023-10-12T22:27:12.121 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:27:12.122 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 59 M 2023-10-12T22:27:12.122 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:27:12.781 INFO:teuthology.orchestra.run.smithi131.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 41 MB/s | 6.0 MB 00:00 2023-10-12T22:27:12.814 INFO:teuthology.orchestra.run.smithi131.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 750 kB/s | 25 kB 00:00 2023-10-12T22:27:12.856 INFO:teuthology.orchestra.run.smithi131.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-10-12T22:27:12.890 INFO:teuthology.orchestra.run.smithi131.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-10-12T22:27:12.999 INFO:teuthology.orchestra.run.smithi131.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 38 MB/s | 4.1 MB 00:00 2023-10-12T22:27:13.024 INFO:teuthology.orchestra.run.smithi131.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-10-12T22:27:13.066 INFO:teuthology.orchestra.run.smithi131.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 1.9 MB/s | 80 kB 00:00 2023-10-12T22:27:13.091 INFO:teuthology.orchestra.run.smithi131.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2023-10-12T22:27:13.125 INFO:teuthology.orchestra.run.smithi131.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-10-12T22:27:13.159 INFO:teuthology.orchestra.run.smithi131.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-10-12T22:27:13.185 INFO:teuthology.orchestra.run.smithi131.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-10-12T22:27:13.343 INFO:teuthology.orchestra.run.smithi131.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-10-12T22:27:13.618 INFO:teuthology.orchestra.run.smithi131.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2023-10-12T22:27:13.702 INFO:teuthology.orchestra.run.smithi131.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 281 kB/s | 145 kB 00:00 2023-10-12T22:27:13.810 INFO:teuthology.orchestra.run.smithi131.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 406 kB/s | 44 kB 00:00 2023-10-12T22:27:13.952 INFO:teuthology.orchestra.run.smithi131.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 298 kB/s | 42 kB 00:00 2023-10-12T22:27:14.002 INFO:teuthology.orchestra.run.smithi131.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 936 kB/s | 47 kB 00:00 2023-10-12T22:27:14.028 INFO:teuthology.orchestra.run.smithi131.stdout:(18/27): librdkafka-0.11.4-3.el8.x86_64.rpm 517 kB/s | 354 kB 00:00 2023-10-12T22:27:14.053 INFO:teuthology.orchestra.run.smithi131.stdout:(19/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.6 MB/s | 80 kB 00:00 2023-10-12T22:27:14.153 INFO:teuthology.orchestra.run.smithi131.stdout:(20/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 485 kB/s | 259 kB 00:00 2023-10-12T22:27:14.278 INFO:teuthology.orchestra.run.smithi131.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 464 kB/s | 116 kB 00:00 2023-10-12T22:27:14.354 INFO:teuthology.orchestra.run.smithi131.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2023-10-12T22:27:14.379 INFO:teuthology.orchestra.run.smithi131.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 586 kB/s | 59 kB 00:00 2023-10-12T22:27:14.404 INFO:teuthology.orchestra.run.smithi131.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 724 kB/s | 181 kB 00:00 2023-10-12T22:27:14.430 INFO:teuthology.orchestra.run.smithi131.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 992 kB/s | 75 kB 00:00 2023-10-12T22:27:14.513 INFO:teuthology.orchestra.run.smithi131.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2023-10-12T22:27:14.580 INFO:teuthology.orchestra.run.smithi131.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 23 MB/s | 4.0 MB 00:00 2023-10-12T22:27:14.586 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:27:14.586 INFO:teuthology.orchestra.run.smithi131.stdout:Total 24 MB/s | 59 MB 00:02 2023-10-12T22:27:14.655 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:27:14.703 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:27:14.704 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:27:15.049 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:27:15.052 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:27:15.766 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:27:15.769 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:27:15.769 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2023-10-12T22:27:15.769 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:27:15.769 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:27:15.770 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-10-12T22:27:15.770 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrading: 2023-10-12T22:27:15.770 INFO:teuthology.orchestra.run.smithi195.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-10-12T22:27:15.771 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-10-12T22:27:15.774 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2023-10-12T22:27:15.774 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-10-12T22:27:15.774 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-10-12T22:27:15.775 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-10-12T22:27:15.775 INFO:teuthology.orchestra.run.smithi195.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-10-12T22:27:15.775 INFO:teuthology.orchestra.run.smithi195.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-10-12T22:27:15.775 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-12T22:27:15.776 INFO:teuthology.orchestra.run.smithi195.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-10-12T22:27:15.776 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-10-12T22:27:15.776 INFO:teuthology.orchestra.run.smithi195.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-12T22:27:15.777 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-10-12T22:27:15.777 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-12T22:27:15.777 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-10-12T22:27:15.777 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-12T22:27:15.778 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-10-12T22:27:15.778 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-12T22:27:15.778 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-12T22:27:15.778 INFO:teuthology.orchestra.run.smithi195.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-10-12T22:27:15.778 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-10-12T22:27:15.779 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-10-12T22:27:15.779 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-10-12T22:27:15.779 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-12T22:27:15.779 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-10-12T22:27:15.779 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-10-12T22:27:15.779 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-10-12T22:27:15.780 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:27:15.780 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:27:15.780 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:27:15.780 INFO:teuthology.orchestra.run.smithi195.stdout:Install 25 Packages 2023-10-12T22:27:15.780 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrade 2 Packages 2023-10-12T22:27:15.781 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:27:15.781 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 59 M 2023-10-12T22:27:15.781 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:27:16.158 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:27:16.408 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-10-12T22:27:16.637 INFO:teuthology.orchestra.run.smithi195.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 40 MB/s | 6.0 MB 00:00 2023-10-12T22:27:16.653 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-10-12T22:27:16.680 INFO:teuthology.orchestra.run.smithi195.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 575 kB/s | 25 kB 00:00 2023-10-12T22:27:16.720 INFO:teuthology.orchestra.run.smithi195.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 20 MB/s | 801 kB 00:00 2023-10-12T22:27:16.754 INFO:teuthology.orchestra.run.smithi195.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-10-12T22:27:16.840 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-10-12T22:27:16.854 INFO:teuthology.orchestra.run.smithi195.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2023-10-12T22:27:16.880 INFO:teuthology.orchestra.run.smithi195.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-10-12T22:27:16.905 INFO:teuthology.orchestra.run.smithi195.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-10-12T22:27:16.930 INFO:teuthology.orchestra.run.smithi195.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-10-12T22:27:17.066 INFO:teuthology.orchestra.run.smithi195.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 387 kB 00:00 2023-10-12T22:27:17.106 INFO:teuthology.orchestra.run.smithi195.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 9.1 MB/s | 368 kB 00:00 2023-10-12T22:27:17.131 INFO:teuthology.orchestra.run.smithi195.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-10-12T22:27:17.265 INFO:teuthology.orchestra.run.smithi195.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 16 MB/s | 13 MB 00:00 2023-10-12T22:27:17.271 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-10-12T22:27:17.473 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-10-12T22:27:17.565 INFO:teuthology.orchestra.run.smithi195.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 22 MB/s | 24 MB 00:01 2023-10-12T22:27:17.576 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-10-12T22:27:17.591 INFO:teuthology.orchestra.run.smithi195.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 316 kB/s | 145 kB 00:00 2023-10-12T22:27:17.699 INFO:teuthology.orchestra.run.smithi195.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 405 kB/s | 44 kB 00:00 2023-10-12T22:27:17.841 INFO:teuthology.orchestra.run.smithi195.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 298 kB/s | 42 kB 00:00 2023-10-12T22:27:17.909 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-10-12T22:27:17.917 INFO:teuthology.orchestra.run.smithi195.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 622 kB/s | 47 kB 00:00 2023-10-12T22:27:17.959 INFO:teuthology.orchestra.run.smithi195.stdout:(18/27): librdkafka-0.11.4-3.el8.x86_64.rpm 511 kB/s | 354 kB 00:00 2023-10-12T22:27:17.992 INFO:teuthology.orchestra.run.smithi195.stdout:(19/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.0 MB/s | 80 kB 00:00 2023-10-12T22:27:18.059 INFO:teuthology.orchestra.run.smithi195.stdout:(20/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 526 kB/s | 259 kB 00:00 2023-10-12T22:27:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-10-12T22:27:18.151 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-10-12T22:27:18.379 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-10-12T22:27:18.493 INFO:teuthology.orchestra.run.smithi195.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 218 kB/s | 116 kB 00:00 2023-10-12T22:27:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-10-12T22:27:18.610 INFO:teuthology.orchestra.run.smithi195.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 496 kB/s | 306 kB 00:00 2023-10-12T22:27:18.635 INFO:teuthology.orchestra.run.smithi195.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 314 kB/s | 181 kB 00:00 2023-10-12T22:27:18.660 INFO:teuthology.orchestra.run.smithi195.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 351 kB/s | 59 kB 00:00 2023-10-12T22:27:18.744 INFO:teuthology.orchestra.run.smithi195.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2023-10-12T22:27:18.769 INFO:teuthology.orchestra.run.smithi195.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 472 kB/s | 75 kB 00:00 2023-10-12T22:27:18.828 INFO:teuthology.orchestra.run.smithi195.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2023-10-12T22:27:18.832 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:27:18.833 INFO:teuthology.orchestra.run.smithi195.stdout:Total 19 MB/s | 59 MB 00:03 2023-10-12T22:27:18.903 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:27:18.951 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:27:18.952 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:27:18.968 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-10-12T22:27:19.148 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-10-12T22:27:19.226 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-10-12T22:27:19.296 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:27:19.299 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:27:19.420 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-10-12T22:27:19.633 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-10-12T22:27:19.700 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-10-12T22:27:19.881 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-10-12T22:27:20.057 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-10-12T22:27:20.248 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-10-12T22:27:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-10-12T22:27:20.453 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:27:20.498 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-10-12T22:27:20.697 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-10-12T22:27:20.919 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-10-12T22:27:20.942 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-10-12T22:27:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-10-12T22:27:21.139 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-10-12T22:27:21.238 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-10-12T22:27:21.460 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-10-12T22:27:21.503 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-10-12T22:27:21.682 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-10-12T22:27:21.726 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-10-12T22:27:21.803 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-10-12T22:27:21.831 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-10-12T22:27:21.902 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-10-12T22:27:22.066 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-10-12T22:27:22.123 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-10-12T22:27:22.289 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-10-12T22:27:22.439 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-10-12T22:27:22.463 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-10-12T22:27:22.508 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-10-12T22:27:22.689 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-10-12T22:27:22.730 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-10-12T22:27:22.905 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-10-12T22:27:23.371 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-10-12T22:27:23.530 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-10-12T22:27:23.609 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-10-12T22:27:23.827 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-10-12T22:27:24.073 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-10-12T22:27:24.139 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-10-12T22:27:24.294 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-10-12T22:27:24.481 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-10-12T22:27:24.655 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-10-12T22:27:24.749 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-10-12T22:27:24.888 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-10-12T22:27:25.320 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-10-12T22:27:25.530 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-10-12T22:27:25.625 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-10-12T22:27:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-10-12T22:27:25.833 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-10-12T22:27:25.939 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-10-12T22:27:26.057 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-10-12T22:27:26.221 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-10-12T22:27:26.307 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-10-12T22:27:26.531 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-10-12T22:27:26.638 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-10-12T22:27:26.713 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-10-12T22:27:26.895 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-10-12T22:27:26.938 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-10-12T22:27:27.043 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-10-12T22:27:27.097 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-10-12T22:27:27.222 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-10-12T22:27:29.729 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-10-12T22:27:29.971 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-10-12T22:27:30.689 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-10-12T22:27:30.996 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-10-12T22:27:31.089 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-10-12T22:27:31.296 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-10-12T22:27:46.035 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-10-12T22:27:46.036 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /sys 2023-10-12T22:27:46.036 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /proc 2023-10-12T22:27:46.036 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /dev 2023-10-12T22:27:46.036 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /run 2023-10-12T22:27:46.036 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /mnt 2023-10-12T22:27:46.037 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /var/tmp 2023-10-12T22:27:46.037 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /home 2023-10-12T22:27:46.037 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /tmp 2023-10-12T22:27:46.037 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /dev 2023-10-12T22:27:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:27:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-10-12T22:27:47.452 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-10-12T22:27:47.452 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-10-12T22:27:47.713 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-10-12T22:27:47.713 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-10-12T22:27:48.418 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-10-12T22:27:48.418 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-10-12T22:27:48.418 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-10-12T22:27:48.418 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-10-12T22:27:48.419 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-10-12T22:27:48.419 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-10-12T22:27:48.419 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-10-12T22:27:48.419 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-10-12T22:27:48.419 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-10-12T22:27:48.420 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-10-12T22:27:48.420 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-10-12T22:27:48.420 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-10-12T22:27:48.420 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-10-12T22:27:48.420 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-10-12T22:27:48.421 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-10-12T22:27:48.421 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-10-12T22:27:48.421 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-10-12T22:27:48.421 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-10-12T22:27:48.421 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-10-12T22:27:48.421 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-10-12T22:27:48.422 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-10-12T22:27:48.422 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-10-12T22:27:48.422 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-10-12T22:27:48.422 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-10-12T22:27:48.423 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-10-12T22:27:48.423 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-10-12T22:27:48.423 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-10-12T22:27:48.423 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-10-12T22:27:48.423 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-10-12T22:27:49.648 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-10-12T22:27:49.648 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:27:49.648 INFO:teuthology.orchestra.run.smithi131.stdout:Upgraded: 2023-10-12T22:27:49.648 INFO:teuthology.orchestra.run.smithi131.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.649 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:27:49.649 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.649 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.649 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.649 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.650 INFO:teuthology.orchestra.run.smithi131.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-10-12T22:27:49.650 INFO:teuthology.orchestra.run.smithi131.stdout: fmt-6.2.1-1.el8.x86_64 2023-10-12T22:27:49.650 INFO:teuthology.orchestra.run.smithi131.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-12T22:27:49.650 INFO:teuthology.orchestra.run.smithi131.stdout: leveldb-1.22-1.el8.x86_64 2023-10-12T22:27:49.651 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.651 INFO:teuthology.orchestra.run.smithi131.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-12T22:27:49.651 INFO:teuthology.orchestra.run.smithi131.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-10-12T22:27:49.651 INFO:teuthology.orchestra.run.smithi131.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-12T22:27:49.651 INFO:teuthology.orchestra.run.smithi131.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.651 INFO:teuthology.orchestra.run.smithi131.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-12T22:27:49.652 INFO:teuthology.orchestra.run.smithi131.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.652 INFO:teuthology.orchestra.run.smithi131.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-12T22:27:49.652 INFO:teuthology.orchestra.run.smithi131.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-12T22:27:49.653 INFO:teuthology.orchestra.run.smithi131.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-10-12T22:27:49.653 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.653 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.654 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.654 INFO:teuthology.orchestra.run.smithi131.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-12T22:27:49.654 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.655 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.655 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:49.656 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:27:49.656 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:27:49.886 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-10-12T22:27:49.886 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /sys 2023-10-12T22:27:49.886 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /proc 2023-10-12T22:27:49.886 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /dev 2023-10-12T22:27:49.887 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /run 2023-10-12T22:27:49.887 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /mnt 2023-10-12T22:27:49.887 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /var/tmp 2023-10-12T22:27:49.887 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /home 2023-10-12T22:27:49.888 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /tmp 2023-10-12T22:27:49.888 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /dev 2023-10-12T22:27:49.888 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:27:49.925 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-test 2023-10-12T22:27:50.775 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:00:43 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:27:51.260 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-10-12T22:27:51.340 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-10-12T22:27:51.340 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-10-12T22:27:51.595 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-10-12T22:27:51.596 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-10-12T22:27:52.332 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-10-12T22:27:52.332 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-10-12T22:27:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-10-12T22:27:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-10-12T22:27:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-10-12T22:27:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-10-12T22:27:52.333 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-10-12T22:27:52.334 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-10-12T22:27:52.334 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-10-12T22:27:52.334 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-10-12T22:27:52.334 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-10-12T22:27:52.334 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-10-12T22:27:52.335 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-10-12T22:27:52.335 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-10-12T22:27:52.335 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-10-12T22:27:52.335 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-10-12T22:27:52.336 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-10-12T22:27:52.336 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-10-12T22:27:52.336 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-10-12T22:27:52.336 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-10-12T22:27:52.337 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-10-12T22:27:52.337 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-10-12T22:27:52.337 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-10-12T22:27:52.337 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-10-12T22:27:52.337 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-10-12T22:27:52.338 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-10-12T22:27:52.338 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-10-12T22:27:52.338 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-10-12T22:27:52.338 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-10-12T22:27:52.382 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:27:52.383 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:27:52.383 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2023-10-12T22:27:52.383 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:27:52.383 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:27:52.383 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-10-12T22:27:52.384 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2023-10-12T22:27:52.384 INFO:teuthology.orchestra.run.smithi131.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-12T22:27:52.384 INFO:teuthology.orchestra.run.smithi131.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-10-12T22:27:52.384 INFO:teuthology.orchestra.run.smithi131.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-12T22:27:52.384 INFO:teuthology.orchestra.run.smithi131.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-12T22:27:52.385 INFO:teuthology.orchestra.run.smithi131.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-12T22:27:52.385 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:27:52.385 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:27:52.385 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:27:52.385 INFO:teuthology.orchestra.run.smithi131.stdout:Install 6 Packages 2023-10-12T22:27:52.386 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:27:52.387 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 53 M 2023-10-12T22:27:52.387 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 229 M 2023-10-12T22:27:52.387 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:27:52.768 INFO:teuthology.orchestra.run.smithi131.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 948 kB/s | 169 kB 00:00 2023-10-12T22:27:53.293 INFO:teuthology.orchestra.run.smithi131.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 74 MB/s | 52 MB 00:00 2023-10-12T22:27:53.318 INFO:teuthology.orchestra.run.smithi131.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 278 kB/s | 202 kB 00:00 2023-10-12T22:27:53.427 INFO:teuthology.orchestra.run.smithi131.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 652 kB/s | 71 kB 00:00 2023-10-12T22:27:53.553 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-10-12T22:27:53.553 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:27:53.553 INFO:teuthology.orchestra.run.smithi195.stdout:Upgraded: 2023-10-12T22:27:53.554 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.554 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:27:53.554 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.554 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.555 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.555 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.555 INFO:teuthology.orchestra.run.smithi195.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-10-12T22:27:53.555 INFO:teuthology.orchestra.run.smithi195.stdout: fmt-6.2.1-1.el8.x86_64 2023-10-12T22:27:53.555 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-12T22:27:53.555 INFO:teuthology.orchestra.run.smithi195.stdout: leveldb-1.22-1.el8.x86_64 2023-10-12T22:27:53.556 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.556 INFO:teuthology.orchestra.run.smithi195.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-12T22:27:53.556 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-10-12T22:27:53.556 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-12T22:27:53.557 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.557 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-12T22:27:53.557 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.557 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-12T22:27:53.557 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-12T22:27:53.557 INFO:teuthology.orchestra.run.smithi195.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-10-12T22:27:53.558 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.558 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.558 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.559 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-12T22:27:53.559 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.559 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.559 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-10-12T22:27:53.559 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:27:53.560 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:27:53.627 INFO:teuthology.orchestra.run.smithi131.stdout:(5/6): oniguruma-6.8.2-2.el8.x86_64.rpm 218 kB/s | 187 kB 00:00 2023-10-12T22:27:53.778 INFO:teuthology.orchestra.run.smithi131.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 667 kB/s | 323 kB 00:00 2023-10-12T22:27:53.779 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:27:53.780 INFO:teuthology.orchestra.run.smithi131.stdout:Total 38 MB/s | 53 MB 00:01 2023-10-12T22:27:53.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-test 2023-10-12T22:27:53.822 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:27:53.843 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:27:53.844 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:27:54.153 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:27:54.156 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:27:54.629 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:27:54.669 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:43 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:27:54.862 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-10-12T22:27:55.070 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-10-12T22:27:55.251 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-12T22:27:55.372 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-12T22:27:55.575 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-10-12T22:27:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-10-12T22:27:56.264 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:27:56.265 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:27:56.265 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2023-10-12T22:27:56.266 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:27:56.266 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:27:56.266 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-10-12T22:27:56.266 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2023-10-12T22:27:56.267 INFO:teuthology.orchestra.run.smithi195.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-12T22:27:56.267 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-10-12T22:27:56.267 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-12T22:27:56.267 INFO:teuthology.orchestra.run.smithi195.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-12T22:27:56.267 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-12T22:27:56.268 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:27:56.268 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:27:56.268 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:27:56.268 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2023-10-12T22:27:56.269 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:27:56.270 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 53 M 2023-10-12T22:27:56.270 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 229 M 2023-10-12T22:27:56.270 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:27:56.539 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 169 kB 00:00 2023-10-12T22:27:57.206 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 71 MB/s | 52 MB 00:00 2023-10-12T22:27:57.231 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 268 kB/s | 202 kB 00:00 2023-10-12T22:27:57.265 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 258 kB/s | 187 kB 00:00 2023-10-12T22:27:57.394 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 545 kB/s | 71 kB 00:00 2023-10-12T22:27:57.673 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 692 kB/s | 323 kB 00:00 2023-10-12T22:27:57.674 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:27:57.675 INFO:teuthology.orchestra.run.smithi195.stdout:Total 38 MB/s | 53 MB 00:01 2023-10-12T22:27:57.719 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:27:57.740 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:27:57.741 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:27:58.055 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:27:58.058 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:27:58.537 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:27:58.819 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-10-12T22:27:59.035 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-10-12T22:27:59.233 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-12T22:27:59.319 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-12T22:27:59.558 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-10-12T22:27:59.692 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-10-12T22:28:00.844 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-10-12T22:28:01.115 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-10-12T22:28:01.400 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-10-12T22:28:01.400 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-10-12T22:28:01.401 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-10-12T22:28:01.401 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-10-12T22:28:01.401 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-10-12T22:28:01.401 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-10-12T22:28:02.124 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-10-12T22:28:02.124 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:02.124 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:28:02.125 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-7.el8.x86_64 2023-10-12T22:28:02.125 INFO:teuthology.orchestra.run.smithi131.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-10-12T22:28:02.125 INFO:teuthology.orchestra.run.smithi131.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-10-12T22:28:02.125 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:02.125 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:28:02.367 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph 2023-10-12T22:28:03.235 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:00:56 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:28:04.527 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-10-12T22:28:04.798 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-10-12T22:28:04.839 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:28:04.843 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:04.843 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2023-10-12T22:28:04.843 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:04.844 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:28:04.844 INFO:teuthology.orchestra.run.smithi131.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-10-12T22:28:04.844 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2023-10-12T22:28:04.844 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-10-12T22:28:04.845 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-10-12T22:28:04.845 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-10-12T22:28:04.845 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-10-12T22:28:04.845 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-10-12T22:28:04.845 INFO:teuthology.orchestra.run.smithi131.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-12T22:28:04.846 INFO:teuthology.orchestra.run.smithi131.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-12T22:28:04.846 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-12T22:28:04.846 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-12T22:28:04.846 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-12T22:28:04.847 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-12T22:28:04.847 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-12T22:28:04.847 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-12T22:28:04.847 INFO:teuthology.orchestra.run.smithi131.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-12T22:28:04.848 INFO:teuthology.orchestra.run.smithi131.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-12T22:28:04.848 INFO:teuthology.orchestra.run.smithi131.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-12T22:28:04.848 INFO:teuthology.orchestra.run.smithi131.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-12T22:28:04.848 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-12T22:28:04.848 INFO:teuthology.orchestra.run.smithi131.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-12T22:28:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-12T22:28:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-12T22:28:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-12T22:28:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-12T22:28:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-12T22:28:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-12T22:28:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-12T22:28:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-12T22:28:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-12T22:28:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-12T22:28:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-12T22:28:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-12T22:28:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:04.851 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:28:04.851 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:04.851 INFO:teuthology.orchestra.run.smithi131.stdout:Install 32 Packages 2023-10-12T22:28:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:04.852 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 31 M 2023-10-12T22:28:04.853 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 111 M 2023-10-12T22:28:04.853 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:28:05.024 INFO:teuthology.orchestra.run.smithi131.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 120 kB/s | 6.3 kB 00:00 2023-10-12T22:28:05.075 INFO:teuthology.orchestra.run.smithi131.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-10-12T22:28:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-10-12T22:28:05.113 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-10-12T22:28:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-10-12T22:28:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-10-12T22:28:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-10-12T22:28:05.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-10-12T22:28:05.125 INFO:teuthology.orchestra.run.smithi131.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2023-10-12T22:28:05.158 INFO:teuthology.orchestra.run.smithi131.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2023-10-12T22:28:05.242 INFO:teuthology.orchestra.run.smithi131.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2023-10-12T22:28:05.476 INFO:teuthology.orchestra.run.smithi131.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 45 MB/s | 18 MB 00:00 2023-10-12T22:28:05.576 INFO:teuthology.orchestra.run.smithi131.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 376 kB/s | 157 kB 00:00 2023-10-12T22:28:05.652 INFO:teuthology.orchestra.run.smithi131.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 95 kB/s | 39 kB 00:00 2023-10-12T22:28:05.676 INFO:teuthology.orchestra.run.smithi131.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 569 kB/s | 57 kB 00:00 2023-10-12T22:28:05.852 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-10-12T22:28:05.853 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:05.853 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:28:05.853 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-7.el8.x86_64 2023-10-12T22:28:05.853 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-10-12T22:28:05.853 INFO:teuthology.orchestra.run.smithi195.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-10-12T22:28:05.854 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:05.854 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:28:05.893 INFO:teuthology.orchestra.run.smithi131.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.1 MB/s | 249 kB 00:00 2023-10-12T22:28:05.919 INFO:teuthology.orchestra.run.smithi131.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 234 kB/s | 103 kB 00:00 2023-10-12T22:28:06.011 INFO:teuthology.orchestra.run.smithi131.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.2 MB/s | 457 kB 00:00 2023-10-12T22:28:06.070 INFO:teuthology.orchestra.run.smithi131.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 3.6 MB/s | 641 kB 00:00 2023-10-12T22:28:06.112 INFO:teuthology.orchestra.run.smithi131.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 566 kB/s | 109 kB 00:00 2023-10-12T22:28:06.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph 2023-10-12T22:28:06.212 INFO:teuthology.orchestra.run.smithi131.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 217 kB/s | 44 kB 00:00 2023-10-12T22:28:06.262 INFO:teuthology.orchestra.run.smithi131.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 963 kB/s | 185 kB 00:00 2023-10-12T22:28:06.288 INFO:teuthology.orchestra.run.smithi131.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.0 MB/s | 173 kB 00:00 2023-10-12T22:28:06.313 INFO:teuthology.orchestra.run.smithi131.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2023-10-12T22:28:06.338 INFO:teuthology.orchestra.run.smithi131.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 351 kB/s | 18 kB 00:00 2023-10-12T22:28:06.363 INFO:teuthology.orchestra.run.smithi131.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 969 kB/s | 49 kB 00:00 2023-10-12T22:28:06.389 INFO:teuthology.orchestra.run.smithi131.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-10-12T22:28:06.422 INFO:teuthology.orchestra.run.smithi131.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.8 MB/s | 384 kB 00:00 2023-10-12T22:28:06.448 INFO:teuthology.orchestra.run.smithi131.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2023-10-12T22:28:06.473 INFO:teuthology.orchestra.run.smithi131.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 372 kB/s | 19 kB 00:00 2023-10-12T22:28:06.498 INFO:teuthology.orchestra.run.smithi131.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 477 kB/s | 24 kB 00:00 2023-10-12T22:28:06.524 INFO:teuthology.orchestra.run.smithi131.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 581 kB/s | 29 kB 00:00 2023-10-12T22:28:06.549 INFO:teuthology.orchestra.run.smithi131.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 549 kB/s | 27 kB 00:00 2023-10-12T22:28:06.574 INFO:teuthology.orchestra.run.smithi131.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2023-10-12T22:28:06.599 INFO:teuthology.orchestra.run.smithi131.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-10-12T22:28:06.625 INFO:teuthology.orchestra.run.smithi131.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2023-10-12T22:28:06.650 INFO:teuthology.orchestra.run.smithi131.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 458 kB/s | 23 kB 00:00 2023-10-12T22:28:06.675 INFO:teuthology.orchestra.run.smithi131.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.9 MB/s | 251 kB 00:00 2023-10-12T22:28:06.679 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:28:06.680 INFO:teuthology.orchestra.run.smithi131.stdout:Total 17 MB/s | 31 MB 00:01 2023-10-12T22:28:06.765 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:28:06.805 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:28:06.805 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:28:07.005 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:56 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:28:07.130 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:28:07.133 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:28:08.241 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:28:08.463 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-12T22:28:08.606 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:28:08.608 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:28:08.609 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2023-10-12T22:28:08.609 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:28:08.609 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:28:08.609 INFO:teuthology.orchestra.run.smithi195.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-10-12T22:28:08.609 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2023-10-12T22:28:08.610 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-10-12T22:28:08.610 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-10-12T22:28:08.610 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-10-12T22:28:08.610 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-10-12T22:28:08.611 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-10-12T22:28:08.611 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-12T22:28:08.611 INFO:teuthology.orchestra.run.smithi195.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-12T22:28:08.611 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-12T22:28:08.611 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-12T22:28:08.612 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-12T22:28:08.612 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-12T22:28:08.612 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-12T22:28:08.612 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-12T22:28:08.612 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-12T22:28:08.612 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-12T22:28:08.613 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-12T22:28:08.613 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-12T22:28:08.613 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-12T22:28:08.613 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-12T22:28:08.613 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-12T22:28:08.613 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-12T22:28:08.614 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-12T22:28:08.614 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-12T22:28:08.614 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-12T22:28:08.614 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-12T22:28:08.614 INFO:teuthology.orchestra.run.smithi195.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-12T22:28:08.615 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-12T22:28:08.615 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-12T22:28:08.615 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-12T22:28:08.615 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-12T22:28:08.615 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-12T22:28:08.615 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:08.616 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:28:08.616 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:28:08.616 INFO:teuthology.orchestra.run.smithi195.stdout:Install 32 Packages 2023-10-12T22:28:08.616 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:08.617 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 31 M 2023-10-12T22:28:08.617 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 111 M 2023-10-12T22:28:08.617 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:28:08.648 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-12T22:28:08.924 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-10-12T22:28:09.172 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-10-12T22:28:09.217 INFO:teuthology.orchestra.run.smithi195.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 136 kB/s | 6.3 kB 00:00 2023-10-12T22:28:09.267 INFO:teuthology.orchestra.run.smithi195.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2023-10-12T22:28:09.343 INFO:teuthology.orchestra.run.smithi195.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2023-10-12T22:28:09.368 INFO:teuthology.orchestra.run.smithi195.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2023-10-12T22:28:09.403 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-10-12T22:28:09.452 INFO:teuthology.orchestra.run.smithi195.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2023-10-12T22:28:09.600 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-10-12T22:28:09.719 INFO:teuthology.orchestra.run.smithi195.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 40 MB/s | 18 MB 00:00 2023-10-12T22:28:09.793 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-10-12T22:28:09.819 INFO:teuthology.orchestra.run.smithi195.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 348 kB/s | 157 kB 00:00 2023-10-12T22:28:09.894 INFO:teuthology.orchestra.run.smithi195.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 88 kB/s | 39 kB 00:00 2023-10-12T22:28:09.919 INFO:teuthology.orchestra.run.smithi195.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 569 kB/s | 57 kB 00:00 2023-10-12T22:28:10.054 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-10-12T22:28:10.136 INFO:teuthology.orchestra.run.smithi195.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 248 kB/s | 103 kB 00:00 2023-10-12T22:28:10.170 INFO:teuthology.orchestra.run.smithi195.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 992 kB/s | 249 kB 00:00 2023-10-12T22:28:10.237 INFO:teuthology.orchestra.run.smithi195.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.3 MB/s | 457 kB 00:00 2023-10-12T22:28:10.261 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-10-12T22:28:10.296 INFO:teuthology.orchestra.run.smithi195.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 3.9 MB/s | 641 kB 00:00 2023-10-12T22:28:10.330 INFO:teuthology.orchestra.run.smithi195.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 687 kB/s | 109 kB 00:00 2023-10-12T22:28:10.490 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-10-12T22:28:10.505 INFO:teuthology.orchestra.run.smithi195.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 163 kB/s | 44 kB 00:00 2023-10-12T22:28:10.647 INFO:teuthology.orchestra.run.smithi195.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 528 kB/s | 185 kB 00:00 2023-10-12T22:28:10.705 INFO:teuthology.orchestra.run.smithi195.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 461 kB/s | 173 kB 00:00 2023-10-12T22:28:10.739 INFO:teuthology.orchestra.run.smithi195.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2023-10-12T22:28:10.761 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-10-12T22:28:10.764 INFO:teuthology.orchestra.run.smithi195.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2023-10-12T22:28:10.789 INFO:teuthology.orchestra.run.smithi195.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 208 kB/s | 18 kB 00:00 2023-10-12T22:28:10.823 INFO:teuthology.orchestra.run.smithi195.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 581 kB/s | 49 kB 00:00 2023-10-12T22:28:10.848 INFO:teuthology.orchestra.run.smithi195.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 700 kB/s | 59 kB 00:00 2023-10-12T22:28:10.898 INFO:teuthology.orchestra.run.smithi195.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 207 kB/s | 16 kB 00:00 2023-10-12T22:28:10.924 INFO:teuthology.orchestra.run.smithi195.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 249 kB/s | 19 kB 00:00 2023-10-12T22:28:10.949 INFO:teuthology.orchestra.run.smithi195.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2023-10-12T22:28:10.974 INFO:teuthology.orchestra.run.smithi195.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 316 kB/s | 24 kB 00:00 2023-10-12T22:28:11.000 INFO:teuthology.orchestra.run.smithi195.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2023-10-12T22:28:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-10-12T22:28:11.050 INFO:teuthology.orchestra.run.smithi195.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 275 kB/s | 27 kB 00:00 2023-10-12T22:28:11.100 INFO:teuthology.orchestra.run.smithi195.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2023-10-12T22:28:11.126 INFO:teuthology.orchestra.run.smithi195.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.0 MB/s | 251 kB 00:00 2023-10-12T22:28:11.150 INFO:teuthology.orchestra.run.smithi195.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 848 kB/s | 85 kB 00:00 2023-10-12T22:28:11.176 INFO:teuthology.orchestra.run.smithi195.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 306 kB/s | 23 kB 00:00 2023-10-12T22:28:11.180 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:28:11.180 INFO:teuthology.orchestra.run.smithi195.stdout:Total 12 MB/s | 31 MB 00:02 2023-10-12T22:28:11.266 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:28:11.304 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:28:11.305 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:28:11.395 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-10-12T22:28:11.632 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:28:11.635 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:28:11.659 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-10-12T22:28:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-10-12T22:28:12.189 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-10-12T22:28:12.390 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-12T22:28:12.742 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-10-12T22:28:12.792 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:28:12.970 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-10-12T22:28:12.986 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-12T22:28:13.183 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-10-12T22:28:13.214 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-12T22:28:13.419 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-12T22:28:13.449 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-10-12T22:28:13.654 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-10-12T22:28:13.679 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-12T22:28:13.877 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-10-12T22:28:13.901 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-12T22:28:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-10-12T22:28:14.157 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-10-12T22:28:14.251 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-10-12T22:28:14.429 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-10-12T22:28:14.515 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-10-12T22:28:14.654 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-10-12T22:28:14.845 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-12T22:28:14.917 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-10-12T22:28:15.114 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-10-12T22:28:15.226 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-10-12T22:28:15.378 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-10-12T22:28:15.463 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-10-12T22:28:15.763 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-10-12T22:28:16.058 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-10-12T22:28:16.295 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-10-12T22:28:16.547 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-10-12T22:28:16.740 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-12T22:28:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-10-12T22:28:17.093 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-10-12T22:28:17.205 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-10-12T22:28:17.328 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-10-12T22:28:17.534 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-10-12T22:28:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-10-12T22:28:17.753 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-12T22:28:17.888 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-10-12T22:28:18.017 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-12T22:28:18.112 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-10-12T22:28:18.227 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-12T22:28:18.297 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-10-12T22:28:18.315 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-10-12T22:28:18.455 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-10-12T22:28:18.480 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-10-12T22:28:18.849 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-10-12T22:28:18.865 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-10-12T22:28:18.865 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-10-12T22:28:18.866 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-10-12T22:28:18.866 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-10-12T22:28:18.866 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-10-12T22:28:18.866 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-10-12T22:28:18.866 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-10-12T22:28:18.867 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-10-12T22:28:18.867 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-10-12T22:28:18.867 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-10-12T22:28:18.867 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-10-12T22:28:18.868 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-10-12T22:28:18.869 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-10-12T22:28:18.870 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-10-12T22:28:18.870 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-10-12T22:28:18.870 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-10-12T22:28:18.870 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-10-12T22:28:18.871 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-12T22:28:18.871 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-10-12T22:28:18.871 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-10-12T22:28:18.871 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-10-12T22:28:18.871 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-10-12T22:28:18.872 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-10-12T22:28:18.872 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-10-12T22:28:18.872 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-10-12T22:28:18.872 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-10-12T22:28:18.872 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-10-12T22:28:18.873 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-10-12T22:28:18.873 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-10-12T22:28:18.873 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-10-12T22:28:18.873 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-10-12T22:28:18.873 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-10-12T22:28:19.163 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-12T22:28:19.553 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-10-12T22:28:19.805 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-10-12T22:28:19.980 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-10-12T22:28:19.980 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:19.981 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:28:19.981 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-10-12T22:28:19.981 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-10-12T22:28:19.981 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-10-12T22:28:19.981 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-10-12T22:28:19.982 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-10-12T22:28:19.982 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-10-12T22:28:19.982 INFO:teuthology.orchestra.run.smithi131.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-12T22:28:19.982 INFO:teuthology.orchestra.run.smithi131.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-12T22:28:19.982 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-12T22:28:19.983 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-12T22:28:19.983 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-12T22:28:19.983 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-12T22:28:19.983 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-12T22:28:19.983 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-12T22:28:19.983 INFO:teuthology.orchestra.run.smithi131.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-12T22:28:19.984 INFO:teuthology.orchestra.run.smithi131.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-12T22:28:19.984 INFO:teuthology.orchestra.run.smithi131.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-12T22:28:19.984 INFO:teuthology.orchestra.run.smithi131.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-12T22:28:19.984 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-12T22:28:19.985 INFO:teuthology.orchestra.run.smithi131.stdout: python3-portend-2.6-1.el8.noarch 2023-10-12T22:28:19.985 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-12T22:28:19.985 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-12T22:28:19.985 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-12T22:28:19.985 INFO:teuthology.orchestra.run.smithi131.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-12T22:28:19.986 INFO:teuthology.orchestra.run.smithi131.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-12T22:28:19.986 INFO:teuthology.orchestra.run.smithi131.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-12T22:28:19.986 INFO:teuthology.orchestra.run.smithi131.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-12T22:28:19.987 INFO:teuthology.orchestra.run.smithi131.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-12T22:28:19.987 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-12T22:28:19.987 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-12T22:28:19.987 INFO:teuthology.orchestra.run.smithi131.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-12T22:28:19.987 INFO:teuthology.orchestra.run.smithi131.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-12T22:28:19.988 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:19.988 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:28:20.265 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-base 2023-10-12T22:28:21.124 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:14 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:28:21.335 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-10-12T22:28:21.538 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-10-12T22:28:21.974 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-10-12T22:28:22.222 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-10-12T22:28:22.445 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-10-12T22:28:22.522 INFO:teuthology.orchestra.run.smithi131.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:28:22.622 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-10-12T22:28:22.640 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-10-12T22:28:22.719 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:28:22.720 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2023-10-12T22:28:22.720 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:28:22.791 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-10-12T22:28:22.829 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install cephadm 2023-10-12T22:28:23.192 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-10-12T22:28:23.192 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-10-12T22:28:23.193 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-10-12T22:28:23.193 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-10-12T22:28:23.193 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-10-12T22:28:23.193 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-10-12T22:28:23.193 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-10-12T22:28:23.194 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-10-12T22:28:23.194 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-10-12T22:28:23.194 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-10-12T22:28:23.194 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-10-12T22:28:23.194 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-10-12T22:28:23.195 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-10-12T22:28:23.195 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-10-12T22:28:23.195 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-10-12T22:28:23.195 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-10-12T22:28:23.195 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-10-12T22:28:23.196 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-12T22:28:23.196 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-10-12T22:28:23.196 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-10-12T22:28:23.196 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-10-12T22:28:23.196 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-10-12T22:28:23.197 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-10-12T22:28:23.197 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-10-12T22:28:23.197 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-10-12T22:28:23.197 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-10-12T22:28:23.197 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-10-12T22:28:23.198 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-10-12T22:28:23.198 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-10-12T22:28:23.199 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-10-12T22:28:23.201 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-10-12T22:28:23.201 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-10-12T22:28:23.690 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:16 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:28:24.292 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-10-12T22:28:24.292 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:24.293 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:28:24.293 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-10-12T22:28:24.293 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-10-12T22:28:24.293 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-10-12T22:28:24.294 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-10-12T22:28:24.294 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-10-12T22:28:24.294 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-10-12T22:28:24.294 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-12T22:28:24.294 INFO:teuthology.orchestra.run.smithi195.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-12T22:28:24.295 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-12T22:28:24.295 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-12T22:28:24.295 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-12T22:28:24.295 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-12T22:28:24.295 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-12T22:28:24.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-12T22:28:24.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-12T22:28:24.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-12T22:28:24.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-12T22:28:24.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-12T22:28:24.297 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-12T22:28:24.297 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend-2.6-1.el8.noarch 2023-10-12T22:28:24.297 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-12T22:28:24.297 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-12T22:28:24.297 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-12T22:28:24.298 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-12T22:28:24.298 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-12T22:28:24.298 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-12T22:28:24.298 INFO:teuthology.orchestra.run.smithi195.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-12T22:28:24.298 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-12T22:28:24.299 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-12T22:28:24.299 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-12T22:28:24.299 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-12T22:28:24.299 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-12T22:28:24.299 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:24.299 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:28:25.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-base 2023-10-12T22:28:25.277 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:28:25.278 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:25.279 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2023-10-12T22:28:25.279 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:25.279 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:28:25.279 INFO:teuthology.orchestra.run.smithi131.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-10-12T22:28:25.279 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:25.280 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:28:25.280 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:25.280 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2023-10-12T22:28:25.280 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:25.280 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 71 k 2023-10-12T22:28:25.281 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 290 k 2023-10-12T22:28:25.281 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:28:25.324 INFO:teuthology.orchestra.run.smithi131.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2023-10-12T22:28:25.325 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:28:25.325 INFO:teuthology.orchestra.run.smithi131.stdout:Total 1.5 MB/s | 71 kB 00:00 2023-10-12T22:28:25.326 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:28:25.332 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:28:25.332 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:28:25.334 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:28:25.337 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:28:25.380 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:28:25.922 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-10-12T22:28:26.001 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:14 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:28:26.100 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-10-12T22:28:26.335 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-10-12T22:28:26.982 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-10-12T22:28:26.983 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:26.983 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:28:26.983 INFO:teuthology.orchestra.run.smithi131.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-10-12T22:28:26.983 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:26.983 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:28:27.210 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-immutable-object-cache 2023-10-12T22:28:27.409 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:28:27.604 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:28:27.605 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2023-10-12T22:28:27.605 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:28:27.725 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install cephadm 2023-10-12T22:28:28.076 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:21 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:28:28.582 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:17 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:28:29.678 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:28:29.679 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:29.679 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repo Size 2023-10-12T22:28:29.679 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:29.680 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:28:29.680 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-10-12T22:28:29.680 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:29.680 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:28:29.680 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:29.681 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2023-10-12T22:28:29.681 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:29.681 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 163 k 2023-10-12T22:28:29.681 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 587 k 2023-10-12T22:28:29.681 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:28:29.739 INFO:teuthology.orchestra.run.smithi131.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 2.7 MB/s | 163 kB 00:00 2023-10-12T22:28:29.740 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:28:29.740 INFO:teuthology.orchestra.run.smithi131.stdout:Total 2.6 MB/s | 163 kB 00:00 2023-10-12T22:28:29.740 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:28:29.750 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:28:29.751 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:28:29.797 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:28:29.800 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:28:29.912 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:28:30.087 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:28:30.188 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:28:30.189 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:28:30.189 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2023-10-12T22:28:30.189 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:28:30.189 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:28:30.189 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-10-12T22:28:30.190 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:30.190 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:28:30.190 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:28:30.190 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2023-10-12T22:28:30.191 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:30.191 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 71 k 2023-10-12T22:28:30.191 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 290 k 2023-10-12T22:28:30.191 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:28:30.241 INFO:teuthology.orchestra.run.smithi195.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2023-10-12T22:28:30.242 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:28:30.242 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.3 MB/s | 71 kB 00:00 2023-10-12T22:28:30.242 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:28:30.249 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:28:30.249 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:28:30.251 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:28:30.254 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:28:30.296 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:28:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:28:31.106 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:28:31.106 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:31.107 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:28:31.107 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-10-12T22:28:31.107 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:31.107 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:28:31.323 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-mgr 2023-10-12T22:28:31.754 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-10-12T22:28:31.925 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-10-12T22:28:32.170 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-10-12T22:28:32.190 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:25 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:28:32.819 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-10-12T22:28:32.819 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:32.820 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:28:32.820 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-10-12T22:28:32.820 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:32.820 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:28:33.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-immutable-object-cache 2023-10-12T22:28:33.599 INFO:teuthology.orchestra.run.smithi131.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:28:33.795 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:28:33.796 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2023-10-12T22:28:33.796 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:28:33.897 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:22 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:28:33.922 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-mgr-dashboard 2023-10-12T22:28:34.782 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:27 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:28:35.481 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:28:35.482 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:28:35.483 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2023-10-12T22:28:35.483 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:28:35.483 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:28:35.483 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-10-12T22:28:35.484 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:35.484 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:28:35.484 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:28:35.484 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2023-10-12T22:28:35.485 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:35.485 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 163 k 2023-10-12T22:28:35.485 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 587 k 2023-10-12T22:28:35.485 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:28:35.522 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 4.2 MB/s | 163 kB 00:00 2023-10-12T22:28:35.523 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:28:35.523 INFO:teuthology.orchestra.run.smithi195.stdout:Total 4.0 MB/s | 163 kB 00:00 2023-10-12T22:28:35.523 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:28:35.534 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:28:35.534 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:28:35.581 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:28:35.584 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:28:35.694 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:28:35.903 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:28:36.311 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:28:36.391 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:28:36.392 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:36.392 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2023-10-12T22:28:36.393 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:36.393 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:28:36.393 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-10-12T22:28:36.393 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2023-10-12T22:28:36.394 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-10-12T22:28:36.394 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-10-12T22:28:36.394 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-12T22:28:36.394 INFO:teuthology.orchestra.run.smithi131.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-12T22:28:36.394 INFO:teuthology.orchestra.run.smithi131.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-12T22:28:36.395 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:36.395 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:28:36.395 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:36.395 INFO:teuthology.orchestra.run.smithi131.stdout:Install 6 Packages 2023-10-12T22:28:36.395 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:36.396 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 3.0 M 2023-10-12T22:28:36.396 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 15 M 2023-10-12T22:28:36.396 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:28:36.835 INFO:teuthology.orchestra.run.smithi131.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 544 kB/s | 24 kB 00:00 2023-10-12T22:28:36.852 INFO:teuthology.orchestra.run.smithi131.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 146 kB/s | 8.9 kB 00:00 2023-10-12T22:28:36.919 INFO:teuthology.orchestra.run.smithi131.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 21 MB/s | 2.7 MB 00:00 2023-10-12T22:28:36.943 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:28:36.944 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:36.944 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:28:36.944 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-10-12T22:28:36.944 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:36.945 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:28:37.011 INFO:teuthology.orchestra.run.smithi131.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 212 kB/s | 34 kB 00:00 2023-10-12T22:28:37.036 INFO:teuthology.orchestra.run.smithi131.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 212 kB/s | 43 kB 00:00 2023-10-12T22:28:37.061 INFO:teuthology.orchestra.run.smithi131.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2023-10-12T22:28:37.064 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:28:37.064 INFO:teuthology.orchestra.run.smithi131.stdout:Total 4.4 MB/s | 3.0 MB 00:00 2023-10-12T22:28:37.074 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:28:37.084 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:28:37.084 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:28:37.258 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:28:37.261 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:28:37.554 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:28:37.791 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-10-12T22:28:38.078 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-10-12T22:28:38.288 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-10-12T22:28:38.425 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-10-12T22:28:38.558 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr 2023-10-12T22:28:39.406 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-10-12T22:28:39.436 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:28 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:28:39.667 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-10-12T22:28:39.952 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-10-12T22:28:39.952 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-10-12T22:28:39.953 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-10-12T22:28:39.953 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-10-12T22:28:39.953 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-10-12T22:28:39.953 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-10-12T22:28:40.600 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-10-12T22:28:40.601 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:40.601 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:28:40.601 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-10-12T22:28:40.601 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-10-12T22:28:40.601 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-10-12T22:28:40.601 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-12T22:28:40.602 INFO:teuthology.orchestra.run.smithi131.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-12T22:28:40.602 INFO:teuthology.orchestra.run.smithi131.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-12T22:28:40.602 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:40.602 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:28:40.745 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-12T22:28:40.839 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:28:41.033 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:28:41.034 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2023-10-12T22:28:41.037 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:28:41.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-dashboard 2023-10-12T22:28:41.601 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:34 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:28:42.014 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:31 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:28:43.186 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:28:43.187 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================================ 2023-10-12T22:28:43.188 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2023-10-12T22:28:43.188 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================================ 2023-10-12T22:28:43.188 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:28:43.188 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-10-12T22:28:43.189 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2023-10-12T22:28:43.189 INFO:teuthology.orchestra.run.smithi131.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-10-12T22:28:43.189 INFO:teuthology.orchestra.run.smithi131.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-10-12T22:28:43.189 INFO:teuthology.orchestra.run.smithi131.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-12T22:28:43.189 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-12T22:28:43.189 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-12T22:28:43.190 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-12T22:28:43.190 INFO:teuthology.orchestra.run.smithi131.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-12T22:28:43.190 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:43.190 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:28:43.190 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================================ 2023-10-12T22:28:43.191 INFO:teuthology.orchestra.run.smithi131.stdout:Install 8 Packages 2023-10-12T22:28:43.191 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:43.191 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 37 M 2023-10-12T22:28:43.192 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 210 M 2023-10-12T22:28:43.192 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:28:43.612 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:28:43.613 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:28:43.613 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2023-10-12T22:28:43.613 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:28:43.614 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:28:43.614 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-10-12T22:28:43.614 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2023-10-12T22:28:43.614 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-10-12T22:28:43.614 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-10-12T22:28:43.615 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-12T22:28:43.615 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-12T22:28:43.615 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-12T22:28:43.615 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:43.615 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:28:43.616 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:28:43.616 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2023-10-12T22:28:43.616 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:43.617 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.0 M 2023-10-12T22:28:43.618 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 15 M 2023-10-12T22:28:43.618 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:28:43.749 INFO:teuthology.orchestra.run.smithi131.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 52 MB/s | 8.6 MB 00:00 2023-10-12T22:28:43.853 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 415 kB/s | 24 kB 00:00 2023-10-12T22:28:43.913 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 76 kB/s | 8.9 kB 00:00 2023-10-12T22:28:43.978 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 14 MB/s | 2.7 MB 00:00 2023-10-12T22:28:44.046 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 251 kB/s | 34 kB 00:00 2023-10-12T22:28:44.079 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 188 kB/s | 43 kB 00:00 2023-10-12T22:28:44.113 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.5 MB/s | 196 kB 00:00 2023-10-12T22:28:44.115 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:28:44.116 INFO:teuthology.orchestra.run.smithi195.stdout:Total 5.9 MB/s | 3.0 MB 00:00 2023-10-12T22:28:44.126 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:28:44.135 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:28:44.135 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:28:44.322 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:28:44.325 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:28:44.501 INFO:teuthology.orchestra.run.smithi131.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 5.2 MB/s | 4.7 MB 00:00 2023-10-12T22:28:44.585 INFO:teuthology.orchestra.run.smithi131.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.9 MB/s | 4.9 MB 00:01 2023-10-12T22:28:44.610 INFO:teuthology.orchestra.run.smithi131.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2023-10-12T22:28:44.611 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:28:44.677 INFO:teuthology.orchestra.run.smithi131.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.0 MB/s | 3.7 MB 00:00 2023-10-12T22:28:44.807 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-10-12T22:28:44.844 INFO:teuthology.orchestra.run.smithi131.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2023-10-12T22:28:44.895 INFO:teuthology.orchestra.run.smithi131.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.2 MB/s | 645 kB 00:00 2023-10-12T22:28:45.078 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-10-12T22:28:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-10-12T22:28:45.441 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-10-12T22:28:45.573 INFO:teuthology.orchestra.run.smithi131.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 14 MB/s | 14 MB 00:00 2023-10-12T22:28:45.574 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:28:45.574 INFO:teuthology.orchestra.run.smithi131.stdout:Total 15 MB/s | 37 MB 00:02 2023-10-12T22:28:45.807 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:28:45.825 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:28:45.825 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:28:46.149 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:28:46.152 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:28:46.436 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-10-12T22:28:46.734 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-10-12T22:28:46.798 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:28:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-12T22:28:47.024 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-10-12T22:28:47.025 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-10-12T22:28:47.025 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-10-12T22:28:47.025 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-10-12T22:28:47.025 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-10-12T22:28:47.025 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-10-12T22:28:47.137 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-12T22:28:47.293 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-12T22:28:47.680 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-10-12T22:28:47.681 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:47.681 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:28:47.681 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-10-12T22:28:47.681 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-10-12T22:28:47.682 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-10-12T22:28:47.682 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-12T22:28:47.682 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-12T22:28:47.682 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-12T22:28:47.682 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:47.683 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:28:47.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-12T22:28:47.897 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-12T22:28:48.076 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-10-12T22:28:48.679 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:37 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:28:48.690 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-10-12T22:28:48.867 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-12T22:28:49.567 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-12T22:28:49.938 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-12T22:28:50.258 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-12T22:28:50.277 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:28:50.278 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================================ 2023-10-12T22:28:50.278 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2023-10-12T22:28:50.279 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================================ 2023-10-12T22:28:50.279 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:28:50.279 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-10-12T22:28:50.279 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2023-10-12T22:28:50.279 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-10-12T22:28:50.279 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-10-12T22:28:50.280 INFO:teuthology.orchestra.run.smithi195.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-12T22:28:50.280 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-12T22:28:50.280 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-12T22:28:50.280 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-12T22:28:50.280 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-12T22:28:50.281 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:50.281 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:28:50.281 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================================ 2023-10-12T22:28:50.281 INFO:teuthology.orchestra.run.smithi195.stdout:Install 8 Packages 2023-10-12T22:28:50.281 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:28:50.282 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 37 M 2023-10-12T22:28:50.282 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 210 M 2023-10-12T22:28:50.282 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:28:50.678 INFO:teuthology.orchestra.run.smithi195.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 57 MB/s | 8.6 MB 00:00 2023-10-12T22:28:51.478 INFO:teuthology.orchestra.run.smithi195.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 5.0 MB/s | 4.7 MB 00:00 2023-10-12T22:28:51.545 INFO:teuthology.orchestra.run.smithi195.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.8 MB/s | 4.9 MB 00:01 2023-10-12T22:28:51.612 INFO:teuthology.orchestra.run.smithi195.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.0 MB/s | 3.7 MB 00:00 2023-10-12T22:28:51.637 INFO:teuthology.orchestra.run.smithi195.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.4 MB/s | 225 kB 00:00 2023-10-12T22:28:51.837 INFO:teuthology.orchestra.run.smithi195.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 862 kB/s | 172 kB 00:00 2023-10-12T22:28:51.937 INFO:teuthology.orchestra.run.smithi195.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.9 MB/s | 645 kB 00:00 2023-10-12T22:28:52.017 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-12T22:28:52.171 INFO:teuthology.orchestra.run.smithi195.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 22 MB/s | 14 MB 00:00 2023-10-12T22:28:52.171 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:28:52.172 INFO:teuthology.orchestra.run.smithi195.stdout:Total 19 MB/s | 37 MB 00:01 2023-10-12T22:28:52.403 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:28:52.421 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:28:52.422 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:28:52.748 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:28:52.751 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:28:53.361 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-10-12T22:28:53.387 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:28:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-10-12T22:28:53.567 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-12T22:28:53.692 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-12T22:28:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-10-12T22:28:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-10-12T22:28:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-10-12T22:28:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-10-12T22:28:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-10-12T22:28:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-10-12T22:28:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-10-12T22:28:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-10-12T22:28:54.553 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-10-12T22:28:54.554 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:54.554 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:28:54.554 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-10-12T22:28:54.554 INFO:teuthology.orchestra.run.smithi131.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-10-12T22:28:54.555 INFO:teuthology.orchestra.run.smithi131.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-10-12T22:28:54.555 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-12T22:28:54.555 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-12T22:28:54.555 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-12T22:28:54.555 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-12T22:28:54.556 INFO:teuthology.orchestra.run.smithi131.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-12T22:28:54.556 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:54.556 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:28:54.676 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-mgr-rook 2023-10-12T22:28:54.792 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-12T22:28:55.390 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-12T22:28:55.528 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:48 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:28:55.574 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-10-12T22:28:56.189 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-10-12T22:28:56.377 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-12T22:28:57.069 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-12T22:28:57.124 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:28:57.126 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:57.126 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2023-10-12T22:28:57.126 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:57.126 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:28:57.126 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-10-12T22:28:57.127 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2023-10-12T22:28:57.127 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-12T22:28:57.127 INFO:teuthology.orchestra.run.smithi131.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-12T22:28:57.127 INFO:teuthology.orchestra.run.smithi131.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-12T22:28:57.128 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-12T22:28:57.128 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-12T22:28:57.128 INFO:teuthology.orchestra.run.smithi131.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-12T22:28:57.128 INFO:teuthology.orchestra.run.smithi131.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-12T22:28:57.128 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-12T22:28:57.128 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-12T22:28:57.129 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-12T22:28:57.129 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-12T22:28:57.129 INFO:teuthology.orchestra.run.smithi131.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-12T22:28:57.129 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:57.129 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:28:57.130 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:28:57.130 INFO:teuthology.orchestra.run.smithi131.stdout:Install 13 Packages 2023-10-12T22:28:57.130 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:28:57.131 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 1.9 M 2023-10-12T22:28:57.131 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 24 M 2023-10-12T22:28:57.131 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:28:57.402 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-12T22:28:57.690 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-12T22:28:57.696 INFO:teuthology.orchestra.run.smithi131.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 545 kB/s | 27 kB 00:00 2023-10-12T22:28:57.896 INFO:teuthology.orchestra.run.smithi131.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 106 kB/s | 27 kB 00:00 2023-10-12T22:28:57.922 INFO:teuthology.orchestra.run.smithi131.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 73 kB/s | 20 kB 00:00 2023-10-12T22:28:58.089 INFO:teuthology.orchestra.run.smithi131.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 573 kB/s | 110 kB 00:00 2023-10-12T22:28:58.114 INFO:teuthology.orchestra.run.smithi131.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 302 kB/s | 126 kB 00:00 2023-10-12T22:28:58.139 INFO:teuthology.orchestra.run.smithi131.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 713 kB/s | 155 kB 00:00 2023-10-12T22:28:58.248 INFO:teuthology.orchestra.run.smithi131.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 268 kB/s | 43 kB 00:00 2023-10-12T22:28:58.273 INFO:teuthology.orchestra.run.smithi131.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2023-10-12T22:28:58.299 INFO:teuthology.orchestra.run.smithi131.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 179 kB/s | 33 kB 00:00 2023-10-12T22:28:58.358 INFO:teuthology.orchestra.run.smithi131.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 745 kB/s | 81 kB 00:00 2023-10-12T22:28:58.424 INFO:teuthology.orchestra.run.smithi131.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 919 kB/s | 61 kB 00:00 2023-10-12T22:28:58.458 INFO:teuthology.orchestra.run.smithi131.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 404 kB/s | 64 kB 00:00 2023-10-12T22:28:58.533 INFO:teuthology.orchestra.run.smithi131.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.5 MB/s | 1.2 MB 00:00 2023-10-12T22:28:58.534 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:28:58.534 INFO:teuthology.orchestra.run.smithi131.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2023-10-12T22:28:58.592 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:28:58.607 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:28:58.608 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:28:58.693 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:28:58.696 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:28:59.177 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:28:59.456 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-12T22:28:59.457 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-12T22:28:59.640 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-10-12T22:28:59.926 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-10-12T22:29:00.164 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-10-12T22:29:00.372 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-10-12T22:29:00.647 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-10-12T22:29:00.789 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-10-12T22:29:00.956 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-10-12T22:29:00.973 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-10-12T22:29:01.255 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-12T22:29:01.290 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-10-12T22:29:01.290 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-10-12T22:29:01.291 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-10-12T22:29:01.291 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-10-12T22:29:01.291 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-10-12T22:29:01.291 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-10-12T22:29:01.291 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-10-12T22:29:01.292 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-10-12T22:29:01.941 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-10-12T22:29:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:01.942 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:29:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-10-12T22:29:01.942 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-10-12T22:29:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-10-12T22:29:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-12T22:29:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-12T22:29:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-12T22:29:01.943 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-12T22:29:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-12T22:29:01.944 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:01.944 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:02.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-rook 2023-10-12T22:29:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-12T22:29:02.688 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-10-12T22:29:02.908 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:51 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:02.914 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-12T22:29:03.164 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-10-12T22:29:03.357 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-10-12T22:29:03.661 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-10-12T22:29:03.662 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-10-12T22:29:03.662 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-10-12T22:29:03.662 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-10-12T22:29:03.662 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-10-12T22:29:03.663 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-10-12T22:29:03.663 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-12T22:29:03.663 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-12T22:29:03.663 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-10-12T22:29:03.664 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-12T22:29:03.664 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-10-12T22:29:03.664 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-10-12T22:29:03.664 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-10-12T22:29:04.388 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-10-12T22:29:04.388 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:04.388 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:29:04.389 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-10-12T22:29:04.389 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-12T22:29:04.389 INFO:teuthology.orchestra.run.smithi131.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-12T22:29:04.389 INFO:teuthology.orchestra.run.smithi131.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-12T22:29:04.389 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-12T22:29:04.390 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-12T22:29:04.390 INFO:teuthology.orchestra.run.smithi131.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-12T22:29:04.390 INFO:teuthology.orchestra.run.smithi131.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-12T22:29:04.390 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-12T22:29:04.390 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-12T22:29:04.391 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-12T22:29:04.391 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-12T22:29:04.391 INFO:teuthology.orchestra.run.smithi131.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-12T22:29:04.391 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:04.391 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:04.515 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:04.517 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:04.517 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2023-10-12T22:29:04.517 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:04.518 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:29:04.518 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-10-12T22:29:04.518 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2023-10-12T22:29:04.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-12T22:29:04.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-12T22:29:04.519 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-12T22:29:04.520 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-12T22:29:04.520 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-12T22:29:04.520 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-12T22:29:04.521 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-12T22:29:04.521 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-12T22:29:04.522 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-12T22:29:04.522 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-12T22:29:04.522 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-12T22:29:04.523 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-12T22:29:04.523 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:04.523 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:29:04.524 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:04.524 INFO:teuthology.orchestra.run.smithi195.stdout:Install 13 Packages 2023-10-12T22:29:04.524 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:04.525 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 1.9 M 2023-10-12T22:29:04.526 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 24 M 2023-10-12T22:29:04.526 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:29:04.528 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-mgr-cephadm 2023-10-12T22:29:04.824 INFO:teuthology.orchestra.run.smithi195.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 463 kB/s | 27 kB 00:00 2023-10-12T22:29:05.005 INFO:teuthology.orchestra.run.smithi195.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 111 kB/s | 27 kB 00:00 2023-10-12T22:29:05.030 INFO:teuthology.orchestra.run.smithi195.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 76 kB/s | 20 kB 00:00 2023-10-12T22:29:05.189 INFO:teuthology.orchestra.run.smithi195.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 598 kB/s | 110 kB 00:00 2023-10-12T22:29:05.214 INFO:teuthology.orchestra.run.smithi195.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 842 kB/s | 155 kB 00:00 2023-10-12T22:29:05.239 INFO:teuthology.orchestra.run.smithi195.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 305 kB/s | 126 kB 00:00 2023-10-12T22:29:05.348 INFO:teuthology.orchestra.run.smithi195.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 268 kB/s | 43 kB 00:00 2023-10-12T22:29:05.365 INFO:teuthology.orchestra.run.smithi195.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 130 kB/s | 16 kB 00:00 2023-10-12T22:29:05.388 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:58 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:05.390 INFO:teuthology.orchestra.run.smithi195.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 188 kB/s | 33 kB 00:00 2023-10-12T22:29:05.449 INFO:teuthology.orchestra.run.smithi195.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 1.1 MB/s | 64 kB 00:00 2023-10-12T22:29:05.508 INFO:teuthology.orchestra.run.smithi195.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 508 kB/s | 81 kB 00:00 2023-10-12T22:29:05.549 INFO:teuthology.orchestra.run.smithi195.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 611 kB/s | 61 kB 00:00 2023-10-12T22:29:05.783 INFO:teuthology.orchestra.run.smithi195.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.8 MB/s | 1.2 MB 00:00 2023-10-12T22:29:05.785 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:05.785 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.5 MB/s | 1.9 MB 00:01 2023-10-12T22:29:05.838 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:29:05.852 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:29:05.853 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:29:05.939 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:29:05.942 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:29:06.459 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:29:06.721 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-12T22:29:06.931 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-10-12T22:29:06.983 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:06.984 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:06.984 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2023-10-12T22:29:06.984 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:06.984 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:29:06.984 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-10-12T22:29:06.985 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2023-10-12T22:29:06.985 INFO:teuthology.orchestra.run.smithi131.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-10-12T22:29:06.985 INFO:teuthology.orchestra.run.smithi131.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-12T22:29:06.985 INFO:teuthology.orchestra.run.smithi131.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-10-12T22:29:06.985 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-12T22:29:06.986 INFO:teuthology.orchestra.run.smithi131.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-10-12T22:29:06.986 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:06.986 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:29:06.986 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:06.987 INFO:teuthology.orchestra.run.smithi131.stdout:Install 6 Packages 2023-10-12T22:29:06.987 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:06.987 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 5.8 M 2023-10-12T22:29:06.988 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 24 M 2023-10-12T22:29:06.988 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:29:07.141 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-10-12T22:29:07.404 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-10-12T22:29:07.584 INFO:teuthology.orchestra.run.smithi131.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.2 MB/s | 89 kB 00:00 2023-10-12T22:29:07.637 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-10-12T22:29:07.867 INFO:teuthology.orchestra.run.smithi131.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 75 kB/s | 21 kB 00:00 2023-10-12T22:29:07.870 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-10-12T22:29:08.188 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-10-12T22:29:08.205 INFO:teuthology.orchestra.run.smithi131.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.0 MB/s | 355 kB 00:00 2023-10-12T22:29:08.235 INFO:teuthology.orchestra.run.smithi131.stdout:(4/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 779 kB/s | 538 kB 00:00 2023-10-12T22:29:08.335 INFO:teuthology.orchestra.run.smithi131.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 376 kB/s | 49 kB 00:00 2023-10-12T22:29:08.410 INFO:teuthology.orchestra.run.smithi131.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.5 MB/s | 4.8 MB 00:00 2023-10-12T22:29:08.412 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:08.412 INFO:teuthology.orchestra.run.smithi131.stdout:Total 4.1 MB/s | 5.8 MB 00:01 2023-10-12T22:29:08.454 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-12T22:29:08.466 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:29:08.477 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:29:08.477 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:29:08.578 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:29:08.581 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:29:08.850 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:29:09.117 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-10-12T22:29:09.392 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-10-12T22:29:09.523 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-12T22:29:09.804 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-10-12T22:29:09.988 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-12T22:29:10.213 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-10-12T22:29:10.385 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-10-12T22:29:10.405 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-10-12T22:29:10.707 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-10-12T22:29:10.708 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-10-12T22:29:10.708 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-10-12T22:29:10.708 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-10-12T22:29:10.708 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-10-12T22:29:10.708 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-10-12T22:29:10.709 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-12T22:29:10.709 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-12T22:29:10.709 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-10-12T22:29:10.709 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-12T22:29:10.709 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-10-12T22:29:10.710 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-10-12T22:29:10.710 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-10-12T22:29:10.765 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-10-12T22:29:11.010 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-10-12T22:29:11.214 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-10-12T22:29:11.413 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-10-12T22:29:11.413 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:11.414 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:29:11.414 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-10-12T22:29:11.414 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-12T22:29:11.414 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-12T22:29:11.414 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-12T22:29:11.415 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-12T22:29:11.415 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-12T22:29:11.415 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-12T22:29:11.415 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-12T22:29:11.416 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-12T22:29:11.416 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-12T22:29:11.416 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-12T22:29:11.416 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-12T22:29:11.416 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-12T22:29:11.417 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:11.417 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:11.499 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-10-12T22:29:11.499 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-10-12T22:29:11.499 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-10-12T22:29:11.499 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-10-12T22:29:11.499 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-10-12T22:29:11.500 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-10-12T22:29:11.550 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-cephadm 2023-10-12T22:29:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-10-12T22:29:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:12.145 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:29:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-10-12T22:29:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-10-12T22:29:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-10-12T22:29:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:12.147 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:12.268 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-fuse 2023-10-12T22:29:12.413 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:01 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:13.132 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:06 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:14.015 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:14.016 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:14.016 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2023-10-12T22:29:14.016 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:14.016 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:29:14.016 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-10-12T22:29:14.017 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2023-10-12T22:29:14.017 INFO:teuthology.orchestra.run.smithi195.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-10-12T22:29:14.017 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-12T22:29:14.017 INFO:teuthology.orchestra.run.smithi195.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-10-12T22:29:14.017 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-12T22:29:14.018 INFO:teuthology.orchestra.run.smithi195.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-10-12T22:29:14.018 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:14.018 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:29:14.018 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:14.018 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2023-10-12T22:29:14.018 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:14.019 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 5.8 M 2023-10-12T22:29:14.019 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 24 M 2023-10-12T22:29:14.020 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:29:14.174 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 89 kB 00:00 2023-10-12T22:29:14.407 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 91 kB/s | 21 kB 00:00 2023-10-12T22:29:14.666 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 984 kB/s | 538 kB 00:00 2023-10-12T22:29:14.717 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.1 MB/s | 355 kB 00:00 2023-10-12T22:29:14.741 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:14.742 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2023-10-12T22:29:14.742 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:14.743 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:29:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-10-12T22:29:14.743 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2023-10-12T22:29:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-12T22:29:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:14.744 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:29:14.744 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:14.744 INFO:teuthology.orchestra.run.smithi131.stdout:Install 2 Packages 2023-10-12T22:29:14.744 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:14.745 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 968 k 2023-10-12T22:29:14.745 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 3.2 M 2023-10-12T22:29:14.745 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:29:14.838 INFO:teuthology.orchestra.run.smithi131.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2023-10-12T22:29:14.947 INFO:teuthology.orchestra.run.smithi131.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 507 kB/s | 83 kB 00:00 2023-10-12T22:29:14.947 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:14.948 INFO:teuthology.orchestra.run.smithi131.stdout:Total 4.6 MB/s | 968 kB 00:00 2023-10-12T22:29:14.958 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:29:14.979 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:29:14.979 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:29:15.034 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.2 MB/s | 4.8 MB 00:00 2023-10-12T22:29:15.059 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): python3-remoto-1.2.1-1.el8.noarch.rpm 125 kB/s | 49 kB 00:00 2023-10-12T22:29:15.059 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:15.060 INFO:teuthology.orchestra.run.smithi195.stdout:Total 5.6 MB/s | 5.8 MB 00:01 2023-10-12T22:29:15.066 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:29:15.070 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:29:15.113 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:29:15.124 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:29:15.124 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:29:15.247 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:29:15.249 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:29:15.250 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:29:15.526 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:29:15.530 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-10-12T22:29:15.704 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-10-12T22:29:15.772 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-10-12T22:29:16.049 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-10-12T22:29:16.101 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-10-12T22:29:16.102 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-10-12T22:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-10-12T22:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-10-12T22:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:16.990 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install librados-devel 2023-10-12T22:29:17.002 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-10-12T22:29:17.401 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-10-12T22:29:17.650 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-10-12T22:29:17.837 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-10-12T22:29:17.854 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:10 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:18.153 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-10-12T22:29:18.153 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-10-12T22:29:18.154 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-10-12T22:29:18.154 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-10-12T22:29:18.154 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-10-12T22:29:18.154 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-10-12T22:29:18.823 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-10-12T22:29:18.824 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:18.824 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:29:18.824 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-10-12T22:29:18.824 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-10-12T22:29:18.825 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-10-12T22:29:18.825 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:18.825 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:18.989 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-fuse 2023-10-12T22:29:19.443 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:19.444 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:19.444 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2023-10-12T22:29:19.444 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:19.444 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:29:19.445 INFO:teuthology.orchestra.run.smithi131.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-10-12T22:29:19.445 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:19.445 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:29:19.445 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:19.445 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2023-10-12T22:29:19.446 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:19.446 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 131 k 2023-10-12T22:29:19.446 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 439 k 2023-10-12T22:29:19.446 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:29:19.493 INFO:teuthology.orchestra.run.smithi131.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 131 kB 00:00 2023-10-12T22:29:19.494 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:19.494 INFO:teuthology.orchestra.run.smithi131.stdout:Total 2.6 MB/s | 131 kB 00:00 2023-10-12T22:29:19.494 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:29:19.499 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:29:19.499 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:29:19.548 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:29:19.551 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:29:19.673 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:29:19.854 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:08 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:19.945 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:20.872 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:20.872 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:20.873 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:29:20.873 INFO:teuthology.orchestra.run.smithi131.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-10-12T22:29:20.873 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:20.873 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:21.074 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install libcephfs2 2023-10-12T22:29:21.460 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:21.460 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:21.461 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2023-10-12T22:29:21.461 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:21.461 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:29:21.461 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-10-12T22:29:21.461 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2023-10-12T22:29:21.462 INFO:teuthology.orchestra.run.smithi195.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-12T22:29:21.462 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:21.462 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:29:21.462 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:21.462 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2023-10-12T22:29:21.463 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:21.463 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 968 k 2023-10-12T22:29:21.463 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 3.2 M 2023-10-12T22:29:21.464 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:29:21.703 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2023-10-12T22:29:21.803 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 521 kB/s | 83 kB 00:00 2023-10-12T22:29:21.804 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:21.804 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.8 MB/s | 968 kB 00:00 2023-10-12T22:29:21.814 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:29:21.835 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:29:21.835 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:29:21.921 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:29:21.924 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:29:21.937 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:14 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:22.114 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:29:22.421 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-10-12T22:29:22.612 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-10-12T22:29:23.035 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-10-12T22:29:23.035 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-10-12T22:29:23.334 INFO:teuthology.orchestra.run.smithi131.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:23.534 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:23.535 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2023-10-12T22:29:23.536 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:23.725 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-10-12T22:29:23.726 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:23.726 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:29:23.726 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-10-12T22:29:23.727 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:23.727 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:23.948 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados-devel 2023-10-12T22:29:24.266 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install libcephfs-devel 2023-10-12T22:29:24.813 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:13 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:25.131 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:18 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:26.437 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:26.438 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:26.438 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2023-10-12T22:29:26.438 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:26.439 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:29:26.439 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-10-12T22:29:26.439 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:26.439 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:29:26.440 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:26.440 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2023-10-12T22:29:26.440 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:26.440 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 131 k 2023-10-12T22:29:26.440 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 439 k 2023-10-12T22:29:26.441 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:29:26.483 INFO:teuthology.orchestra.run.smithi195.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 131 kB 00:00 2023-10-12T22:29:26.483 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:26.484 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.8 MB/s | 131 kB 00:00 2023-10-12T22:29:26.484 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:29:26.489 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:29:26.489 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:29:26.538 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:29:26.541 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:29:26.647 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:29:26.717 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:26.718 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:26.718 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2023-10-12T22:29:26.719 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:26.719 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:29:26.719 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-10-12T22:29:26.723 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:26.723 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:29:26.723 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:26.723 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2023-10-12T22:29:26.724 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:26.724 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 24 k 2023-10-12T22:29:26.725 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 94 k 2023-10-12T22:29:26.725 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:29:26.767 INFO:teuthology.orchestra.run.smithi131.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 512 kB/s | 24 kB 00:00 2023-10-12T22:29:26.767 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:26.768 INFO:teuthology.orchestra.run.smithi131.stdout:Total 499 kB/s | 24 kB 00:00 2023-10-12T22:29:26.768 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:29:26.771 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:29:26.771 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:29:26.773 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:29:26.777 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:29:26.823 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:29:26.885 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:27.026 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:27.179 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:27.288 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:27.830 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:27.830 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:27.830 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:29:27.830 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-10-12T22:29:27.831 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:27.831 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:27.900 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:29:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-10-12T22:29:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:27.901 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:28.048 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install librados2 2023-10-12T22:29:28.071 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs2 2023-10-12T22:29:28.911 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:21 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:28.937 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:17 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:30.321 INFO:teuthology.orchestra.run.smithi131.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:30.344 INFO:teuthology.orchestra.run.smithi195.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:30.516 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:30.517 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2023-10-12T22:29:30.517 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:30.540 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:30.541 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2023-10-12T22:29:30.542 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:30.638 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install librbd1 2023-10-12T22:29:30.662 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs-devel 2023-10-12T22:29:31.504 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:24 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:31.522 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:20 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:32.906 INFO:teuthology.orchestra.run.smithi131.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:33.100 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:33.101 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2023-10-12T22:29:33.102 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:33.127 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:33.128 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:33.128 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2023-10-12T22:29:33.129 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:33.129 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:29:33.129 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-10-12T22:29:33.129 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:33.129 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:29:33.130 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:33.130 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2023-10-12T22:29:33.130 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:33.131 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 24 k 2023-10-12T22:29:33.131 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 94 k 2023-10-12T22:29:33.131 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:29:33.188 INFO:teuthology.orchestra.run.smithi195.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 412 kB/s | 24 kB 00:00 2023-10-12T22:29:33.189 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:33.189 INFO:teuthology.orchestra.run.smithi195.stdout:Total 403 kB/s | 24 kB 00:00 2023-10-12T22:29:33.190 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:29:33.193 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:29:33.193 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:29:33.195 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:29:33.198 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:29:33.221 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install python3-rados 2023-10-12T22:29:33.247 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:29:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:33.755 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:34.084 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:27 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:34.399 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:34.399 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:34.399 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:29:34.399 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-10-12T22:29:34.399 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:34.400 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:34.537 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados2 2023-10-12T22:29:35.403 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:24 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:35.478 INFO:teuthology.orchestra.run.smithi131.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:35.673 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:35.674 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2023-10-12T22:29:35.675 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:35.790 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install python3-rgw 2023-10-12T22:29:36.652 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:29 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:36.808 INFO:teuthology.orchestra.run.smithi195.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:37.003 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:37.004 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2023-10-12T22:29:37.004 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:37.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librbd1 2023-10-12T22:29:37.987 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:26 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:38.065 INFO:teuthology.orchestra.run.smithi131.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:38.258 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:38.259 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2023-10-12T22:29:38.259 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:38.373 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install python3-cephfs 2023-10-12T22:29:39.239 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:32 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:39.388 INFO:teuthology.orchestra.run.smithi195.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:39.581 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:39.582 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2023-10-12T22:29:39.582 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:39.691 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rados 2023-10-12T22:29:40.552 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:29 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:40.648 INFO:teuthology.orchestra.run.smithi131.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:40.843 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:40.844 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2023-10-12T22:29:40.844 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:40.947 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install python3-rbd 2023-10-12T22:29:41.812 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:34 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:41.957 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:42.151 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:42.152 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2023-10-12T22:29:42.152 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:42.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rgw 2023-10-12T22:29:43.187 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:32 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:43.223 INFO:teuthology.orchestra.run.smithi131.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:43.419 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:43.420 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2023-10-12T22:29:43.421 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:43.569 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install rbd-fuse 2023-10-12T22:29:44.430 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:37 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:44.592 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:44.792 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:44.793 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2023-10-12T22:29:44.793 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:44.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-cephfs 2023-10-12T22:29:45.763 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:34 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:46.024 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:46.025 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:46.025 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2023-10-12T22:29:46.026 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:46.026 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:29:46.026 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-10-12T22:29:46.026 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:46.026 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:29:46.027 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:46.027 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2023-10-12T22:29:46.027 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:46.027 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 89 k 2023-10-12T22:29:46.027 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 298 k 2023-10-12T22:29:46.028 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:29:46.073 INFO:teuthology.orchestra.run.smithi131.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2023-10-12T22:29:46.074 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:46.074 INFO:teuthology.orchestra.run.smithi131.stdout:Total 1.8 MB/s | 89 kB 00:00 2023-10-12T22:29:46.075 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:29:46.084 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:29:46.084 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:29:46.133 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:29:46.136 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:29:46.247 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:29:46.490 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:46.793 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:47.176 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:47.368 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:47.369 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2023-10-12T22:29:47.370 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:47.411 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:47.412 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:47.412 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:29:47.412 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-10-12T22:29:47.412 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:47.413 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:47.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rbd 2023-10-12T22:29:47.644 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install rbd-mirror 2023-10-12T22:29:48.335 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:37 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:48.510 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:41 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:49.750 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-10-12T22:29:49.943 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:49.944 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2023-10-12T22:29:49.944 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:50.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-fuse 2023-10-12T22:29:50.119 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:50.120 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:50.120 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2023-10-12T22:29:50.120 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:50.120 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:29:50.120 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-10-12T22:29:50.121 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:50.121 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:29:50.121 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:50.121 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2023-10-12T22:29:50.121 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:50.122 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 3.7 M 2023-10-12T22:29:50.122 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 18 M 2023-10-12T22:29:50.122 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:29:50.219 INFO:teuthology.orchestra.run.smithi131.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 38 MB/s | 3.7 MB 00:00 2023-10-12T22:29:50.219 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:50.220 INFO:teuthology.orchestra.run.smithi131.stdout:Total 37 MB/s | 3.7 MB 00:00 2023-10-12T22:29:50.220 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:29:50.231 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:29:50.232 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:29:50.295 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:29:50.298 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:29:50.765 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:29:50.927 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:39 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:50.974 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:51.400 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:52.025 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:52.026 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:52.026 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:29:52.026 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-10-12T22:29:52.026 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:52.027 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:52.238 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install rbd-nbd 2023-10-12T22:29:52.536 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:52.537 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:52.538 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2023-10-12T22:29:52.538 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:52.538 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:29:52.538 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-10-12T22:29:52.538 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:52.539 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:29:52.539 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:52.539 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2023-10-12T22:29:52.539 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:52.540 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 89 k 2023-10-12T22:29:52.540 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 298 k 2023-10-12T22:29:52.540 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:29:52.587 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2023-10-12T22:29:52.588 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:52.588 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.7 MB/s | 89 kB 00:00 2023-10-12T22:29:52.588 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:29:52.597 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:29:52.598 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:29:52.648 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:29:52.651 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:29:52.770 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:29:52.998 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:53.114 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:02:46 ago on Thu 12 Oct 2023 10:27:07 PM UTC. 2023-10-12T22:29:53.296 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:54.040 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:54.040 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:54.040 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:29:54.041 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-10-12T22:29:54.041 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:54.041 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:54.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-mirror 2023-10-12T22:29:54.718 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-10-12T22:29:54.719 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:54.719 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2023-10-12T22:29:54.720 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:54.720 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-10-12T22:29:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-10-12T22:29:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:54.720 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-10-12T22:29:54.721 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-10-12T22:29:54.721 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2023-10-12T22:29:54.721 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:54.721 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 173 k 2023-10-12T22:29:54.721 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 649 k 2023-10-12T22:29:54.722 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-10-12T22:29:54.764 INFO:teuthology.orchestra.run.smithi131.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.9 MB/s | 173 kB 00:00 2023-10-12T22:29:54.765 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:54.765 INFO:teuthology.orchestra.run.smithi131.stdout:Total 3.8 MB/s | 173 kB 00:00 2023-10-12T22:29:54.765 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-10-12T22:29:54.775 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-10-12T22:29:54.775 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-10-12T22:29:54.825 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-10-12T22:29:54.827 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-10-12T22:29:54.965 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-10-12T22:29:55.111 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:44 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:29:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:55.456 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:56.137 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:56.138 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:56.138 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-10-12T22:29:56.138 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-10-12T22:29:56.138 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-10-12T22:29:56.138 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-10-12T22:29:56.364 DEBUG:teuthology.parallel:result is None 2023-10-12T22:29:56.689 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:29:56.689 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:56.690 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2023-10-12T22:29:56.690 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:56.690 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:29:56.690 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-10-12T22:29:56.691 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:56.691 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:29:56.691 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:29:56.691 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2023-10-12T22:29:56.692 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:56.692 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.7 M 2023-10-12T22:29:56.692 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 18 M 2023-10-12T22:29:56.692 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:29:56.823 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2023-10-12T22:29:56.824 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:29:56.824 INFO:teuthology.orchestra.run.smithi195.stdout:Total 28 MB/s | 3.7 MB 00:00 2023-10-12T22:29:56.825 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:29:56.836 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:29:56.837 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:29:56.900 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:29:56.903 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:29:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:29:57.615 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:58.041 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:58.676 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:29:58.677 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:58.677 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:29:58.677 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-10-12T22:29:58.677 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:29:58.678 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:29:59.023 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-nbd 2023-10-12T22:29:59.895 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:48 ago on Thu 12 Oct 2023 10:27:11 PM UTC. 2023-10-12T22:30:01.507 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2023-10-12T22:30:01.507 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:30:01.507 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2023-10-12T22:30:01.508 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:30:01.508 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2023-10-12T22:30:01.508 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-10-12T22:30:01.508 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:30:01.508 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2023-10-12T22:30:01.509 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2023-10-12T22:30:01.509 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2023-10-12T22:30:01.509 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:30:01.510 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 173 k 2023-10-12T22:30:01.510 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 649 k 2023-10-12T22:30:01.510 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2023-10-12T22:30:01.577 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 2.5 MB/s | 173 kB 00:00 2023-10-12T22:30:01.578 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2023-10-12T22:30:01.578 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.4 MB/s | 173 kB 00:00 2023-10-12T22:30:01.578 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2023-10-12T22:30:01.588 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2023-10-12T22:30:01.588 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2023-10-12T22:30:01.638 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2023-10-12T22:30:01.641 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2023-10-12T22:30:01.764 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2023-10-12T22:30:01.989 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:30:02.295 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:30:02.966 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-10-12T22:30:02.966 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:30:02.966 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2023-10-12T22:30:02.966 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-10-12T22:30:02.967 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:30:02.967 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2023-10-12T22:30:03.201 DEBUG:teuthology.parallel:result is None 2023-10-12T22:30:03.202 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-10-12T22:30:03.203 INFO:teuthology.packaging:ref: None 2023-10-12T22:30:03.203 INFO:teuthology.packaging:tag: v16.2.4 2023-10-12T22:30:03.204 INFO:teuthology.packaging:branch: None 2023-10-12T22:30:03.204 INFO:teuthology.packaging:sha1: 3da7aaee8cb8051dac57683eeb2f18b9da99d2c9 2023-10-12T22:30:03.400 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-10-12T22:30:03.415 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-10-12T22:30:03.642 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-12T22:30:03.670 INFO:teuthology.orchestra.run.smithi131.stdout:16.2.4-0.el8 2023-10-12T22:30:03.671 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-10-12T22:30:03.671 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-10-12T22:30:03.671 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-10-12T22:30:03.672 INFO:teuthology.packaging:ref: None 2023-10-12T22:30:03.672 INFO:teuthology.packaging:tag: v16.2.4 2023-10-12T22:30:03.672 INFO:teuthology.packaging:branch: None 2023-10-12T22:30:03.673 INFO:teuthology.packaging:sha1: 3da7aaee8cb8051dac57683eeb2f18b9da99d2c9 2023-10-12T22:30:03.868 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-10-12T22:30:03.869 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-10-12T22:30:04.098 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-12T22:30:04.123 INFO:teuthology.orchestra.run.smithi195.stdout:16.2.4-0.el8 2023-10-12T22:30:04.124 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-10-12T22:30:04.124 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-10-12T22:30:04.125 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-12T22:30:04.126 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-10-12T22:30:04.127 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-12T22:30:04.161 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-12T22:30:04.161 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-12T22:30:04.194 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-12T22:30:04.194 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-10-12T22:30:04.195 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/daemon-helper 2023-10-12T22:30:04.231 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-12T22:30:04.302 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-12T22:30:04.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2023-10-12T22:30:04.332 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-12T22:30:04.400 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-12T22:30:04.401 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-10-12T22:30:04.401 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-12T22:30:04.431 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-12T22:30:04.500 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-12T22:30:04.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-12T22:30:04.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-12T22:30:04.596 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-12T22:30:04.597 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-10-12T22:30:04.597 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/stdin-killer 2023-10-12T22:30:04.626 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-12T22:30:04.694 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-12T22:30:04.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/stdin-killer 2023-10-12T22:30:04.722 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-12T22:30:04.789 INFO:teuthology.run_tasks:Running task print... 2023-10-12T22:30:04.811 INFO:teuthology.task.print:**** done install task... 2023-10-12T22:30:04.811 INFO:teuthology.run_tasks:Running task cephadm... 2023-10-12T22:30:04.905 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '3da7aaee8cb8051dac57683eeb2f18b9da99d2c9'} 2023-10-12T22:30:04.906 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-10-12T22:30:04.906 INFO:tasks.cephadm:Cluster fsid is e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:30:04.907 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-10-12T22:30:04.907 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-10-12T22:30:04.907 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi131': '172.21.15.131', 'mon.smithi195': '172.21.15.195'} 2023-10-12T22:30:04.908 INFO:tasks.cephadm:Normalizing hostnames... 2023-10-12T22:30:04.908 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname $(hostname -s) 2023-10-12T22:30:04.937 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname $(hostname -s) 2023-10-12T22:30:04.965 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-10-12T22:30:04.966 DEBUG:teuthology.orchestra.run.smithi131:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-10-12T22:30:05.231 INFO:teuthology.orchestra.run.smithi131.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Oct 12 22:30 /home/ubuntu/cephtest/cephadm 2023-10-12T22:30:05.232 DEBUG:teuthology.orchestra.run.smithi195:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-10-12T22:30:05.500 INFO:teuthology.orchestra.run.smithi195.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Oct 12 22:30 /home/ubuntu/cephtest/cephadm 2023-10-12T22:30:05.501 DEBUG:teuthology.orchestra.run.smithi131:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-10-12T22:30:05.521 DEBUG:teuthology.orchestra.run.smithi195:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-10-12T22:30:05.551 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /etc/ceph 2023-10-12T22:30:05.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph 2023-10-12T22:30:05.617 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 777 /etc/ceph 2023-10-12T22:30:05.658 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /etc/ceph 2023-10-12T22:30:05.685 INFO:tasks.cephadm:Writing seed config... 2023-10-12T22:30:05.686 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-10-12T22:30:05.686 INFO:tasks.cephadm: override: [client] debug client = 20 2023-10-12T22:30:05.687 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-10-12T22:30:05.687 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-10-12T22:30:05.687 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-10-12T22:30:05.688 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-10-12T22:30:05.688 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-10-12T22:30:05.688 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-10-12T22:30:05.689 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-10-12T22:30:05.689 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-10-12T22:30:05.689 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-10-12T22:30:05.689 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-10-12T22:30:05.690 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-10-12T22:30:05.690 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-10-12T22:30:05.690 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-10-12T22:30:05.691 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-10-12T22:30:05.691 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-10-12T22:30:05.691 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-10-12T22:30:05.691 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-10-12T22:30:05.692 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-10-12T22:30:05.692 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-10-12T22:30:05.692 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-10-12T22:30:05.693 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-10-12T22:30:05.693 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-10-12T22:30:05.693 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-10-12T22:30:05.694 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-10-12T22:30:05.694 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-10-12T22:30:05.694 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-10-12T22:30:05.694 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-10-12T22:30:05.695 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-10-12T22:30:05.697 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-10-12T22:30:05.697 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-10-12T22:30:05.697 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-10-12T22:30:05.698 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-10-12T22:30:05.698 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-10-12T22:30:05.698 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-10-12T22:30:05.699 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-10-12T22:30:05.699 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-10-12T22:30:05.700 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-10-12T22:30:05.719 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e4069eba-694e-11ee-8db6-212e2dc638e7 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-10-12T22:30:05.720 DEBUG:teuthology.orchestra.run.smithi131:mon.smithi131> sudo journalctl -f -n 0 -u ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi131.service 2023-10-12T22:30:05.762 INFO:tasks.cephadm:Bootstrapping... 2023-10-12T22:30:05.763 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.131 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-10-12T22:30:05.789 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:-- Logs begin at Thu 2023-10-12 22:18:00 UTC. -- 2023-10-12T22:30:06.043 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-10-12T22:30:06.879 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: 4.6.1 2023-10-12T22:30:06.890 INFO:teuthology.orchestra.run.smithi131.stderr:Verifying podman|docker is present... 2023-10-12T22:30:06.890 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-10-12T22:30:06.913 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: 4.6.1 2023-10-12T22:30:06.924 INFO:teuthology.orchestra.run.smithi131.stderr:Verifying lvm2 is present... 2023-10-12T22:30:06.925 INFO:teuthology.orchestra.run.smithi131.stderr:Verifying time synchronization is in place... 2023-10-12T22:30:06.925 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-enabled chrony.service 2023-10-12T22:30:06.933 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-10-12T22:30:06.934 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-active chrony.service 2023-10-12T22:30:06.940 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: inactive 2023-10-12T22:30:06.940 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-enabled chronyd.service 2023-10-12T22:30:06.946 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: enabled 2023-10-12T22:30:06.947 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-active chronyd.service 2023-10-12T22:30:06.953 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: active 2023-10-12T22:30:06.954 INFO:teuthology.orchestra.run.smithi131.stderr:Unit chronyd.service is enabled and running 2023-10-12T22:30:06.954 INFO:teuthology.orchestra.run.smithi131.stderr:Repeating the final host check... 2023-10-12T22:30:06.954 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-10-12T22:30:06.974 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: 4.6.1 2023-10-12T22:30:06.985 INFO:teuthology.orchestra.run.smithi131.stderr:podman|docker (/bin/podman) is present 2023-10-12T22:30:06.985 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl is present 2023-10-12T22:30:06.985 INFO:teuthology.orchestra.run.smithi131.stderr:lvcreate is present 2023-10-12T22:30:06.985 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-enabled chrony.service 2023-10-12T22:30:06.991 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-10-12T22:30:06.991 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-active chrony.service 2023-10-12T22:30:06.997 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: inactive 2023-10-12T22:30:06.997 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-enabled chronyd.service 2023-10-12T22:30:07.003 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: enabled 2023-10-12T22:30:07.004 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-active chronyd.service 2023-10-12T22:30:07.009 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: active 2023-10-12T22:30:07.010 INFO:teuthology.orchestra.run.smithi131.stderr:Unit chronyd.service is enabled and running 2023-10-12T22:30:07.010 INFO:teuthology.orchestra.run.smithi131.stderr:Host looks OK 2023-10-12T22:30:07.010 INFO:teuthology.orchestra.run.smithi131.stderr:Cluster fsid: e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:30:07.011 INFO:teuthology.orchestra.run.smithi131.stderr:Acquiring lock 140242855250408 on /run/cephadm/e4069eba-694e-11ee-8db6-212e2dc638e7.lock 2023-10-12T22:30:07.011 INFO:teuthology.orchestra.run.smithi131.stderr:Lock 140242855250408 acquired on /run/cephadm/e4069eba-694e-11ee-8db6-212e2dc638e7.lock 2023-10-12T22:30:07.012 INFO:teuthology.orchestra.run.smithi131.stderr:Verifying IP 172.21.15.131 port 3300 ... 2023-10-12T22:30:07.012 INFO:teuthology.orchestra.run.smithi131.stderr:Verifying IP 172.21.15.131 port 6789 ... 2023-10-12T22:30:07.012 INFO:teuthology.orchestra.run.smithi131.stderr:Base mon IP is 172.21.15.131, final addrv is [v2:172.21.15.131:3300,v1:172.21.15.131:6789] 2023-10-12T22:30:07.012 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /sbin/ip route ls 2023-10-12T22:30:07.013 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.131 metric 100 2023-10-12T22:30:07.014 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.131 metric 100 2023-10-12T22:30:07.014 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /sbin/ip -6 route ls 2023-10-12T22:30:07.016 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-10-12T22:30:07.017 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-10-12T22:30:07.017 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2023-10-12T22:30:07.017 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /sbin/ip -6 addr ls 2023-10-12T22:30:07.019 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-10-12T22:30:07.020 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-10-12T22:30:07.020 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-10-12T22:30:07.020 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-10-12T22:30:07.020 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:72f9/64 scope link 2023-10-12T22:30:07.020 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-10-12T22:30:07.021 INFO:teuthology.orchestra.run.smithi131.stderr:Mon IP 172.21.15.131 is in CIDR network 172.21.0.0/20 2023-10-12T22:30:07.021 INFO:teuthology.orchestra.run.smithi131.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-10-12T22:30:07.022 INFO:teuthology.orchestra.run.smithi131.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-10-12T22:30:07.022 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-10-12T22:30:07.119 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-10-12T22:30:13.699 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Getting image source signatures 2023-10-12T22:30:13.704 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-10-12T22:30:13.705 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-10-12T22:30:26.037 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-10-12T22:30:26.087 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Writing manifest to image destination 2023-10-12T22:30:26.398 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-10-12T22:30:26.409 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-10-12T22:30:26.954 INFO:teuthology.orchestra.run.smithi131.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-10-12T22:30:27.325 INFO:teuthology.orchestra.run.smithi131.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-10-12T22:30:27.325 INFO:teuthology.orchestra.run.smithi131.stderr:Extracting ceph user uid/gid from container image... 2023-10-12T22:30:27.325 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-10-12T22:30:27.584 INFO:teuthology.orchestra.run.smithi131.stderr:stat: 167 167 2023-10-12T22:30:27.913 INFO:teuthology.orchestra.run.smithi131.stderr:Creating initial keys... 2023-10-12T22:30:27.913 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-10-12T22:30:28.175 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-authtool: AQCEcyhlCx4gChAAoJKEXF+6tBm0Um9rWETXpQ== 2023-10-12T22:30:28.506 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-10-12T22:30:28.766 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-authtool: AQCEcyhl1ZniLBAAr+H5BriDrYMoo//ChdPWBQ== 2023-10-12T22:30:29.084 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-10-12T22:30:29.325 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-authtool: AQCFcyhlRKHcEhAAYYfxwdUbdv8G8DpLRa0agA== 2023-10-12T22:30:29.657 INFO:teuthology.orchestra.run.smithi131.stderr:Creating initial monmap... 2023-10-12T22:30:29.657 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpu0_fhq2l:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 --addv smithi131 [v2:172.21.15.131:3300,v1:172.21.15.131:6789] /tmp/monmap 2023-10-12T22:30:29.909 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-10-12T22:30:29.909 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:30:29.909 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-10-12T22:30:30.254 INFO:teuthology.orchestra.run.smithi131.stderr:monmaptool for smithi131 [v2:172.21.15.131:3300,v1:172.21.15.131:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-10-12T22:30:30.255 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/monmaptool: set fsid to e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:30:30.255 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-10-12T22:30:30.255 INFO:teuthology.orchestra.run.smithi131.stderr: 2023-10-12T22:30:30.255 INFO:teuthology.orchestra.run.smithi131.stderr:Creating mon... 2023-10-12T22:30:30.256 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/mon.smithi131:/var/lib/ceph/mon/ceph-smithi131:z -v /tmp/ceph-tmpecnb_qud:/tmp/keyring:z -v /tmp/ceph-tmpu0_fhq2l:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi131 --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-10-12T22:30:30.541 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.514+0000 7fc114bba700 0 set uid:gid to 167:167 (ceph:ceph) 2023-10-12T22:30:30.541 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.515+0000 7fc114bba700 1 imported monmap: 2023-10-12T22:30:30.541 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: epoch 0 2023-10-12T22:30:30.542 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: fsid e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:30:30.542 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: last_changed 2023-10-12T22:30:29.908459+0000 2023-10-12T22:30:30.542 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: created 2023-10-12T22:30:29.908459+0000 2023-10-12T22:30:30.542 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-10-12T22:30:30.542 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-10-12T22:30:30.543 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon.smithi131 2023-10-12T22:30:30.543 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.543 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.515+0000 7fc114bba700 0 /usr/bin/ceph-mon: set fsid to e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:30:30.543 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: RocksDB version: 6.8.1 2023-10-12T22:30:30.543 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.543 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-10-12T22:30:30.544 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Compile date May 13 2021 2023-10-12T22:30:30.544 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: DB SUMMARY 2023-10-12T22:30:30.544 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.544 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi131/store.db dir, Total Num: 0, files: 2023-10-12T22:30:30.544 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.544 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi131/store.db: 2023-10-12T22:30:30.545 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.545 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.error_if_exists: 0 2023-10-12T22:30:30.545 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.create_if_missing: 1 2023-10-12T22:30:30.545 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.paranoid_checks: 1 2023-10-12T22:30:30.545 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.env: 0x5615716a71c0 2023-10-12T22:30:30.546 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.fs: Posix File System 2023-10-12T22:30:30.546 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.info_log: 0x56157368eec0 2023-10-12T22:30:30.546 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_file_opening_threads: 16 2023-10-12T22:30:30.546 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.statistics: (nil) 2023-10-12T22:30:30.546 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.use_fsync: 0 2023-10-12T22:30:30.547 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_log_file_size: 0 2023-10-12T22:30:30.547 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-10-12T22:30:30.547 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-10-12T22:30:30.547 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.keep_log_file_num: 1000 2023-10-12T22:30:30.547 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.recycle_log_file_num: 0 2023-10-12T22:30:30.547 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.allow_fallocate: 1 2023-10-12T22:30:30.548 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.allow_mmap_reads: 0 2023-10-12T22:30:30.548 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.allow_mmap_writes: 0 2023-10-12T22:30:30.548 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.use_direct_reads: 0 2023-10-12T22:30:30.548 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-10-12T22:30:30.548 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.create_missing_column_families: 0 2023-10-12T22:30:30.548 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.db_log_dir: 2023-10-12T22:30:30.549 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi131/store.db 2023-10-12T22:30:30.549 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-10-12T22:30:30.549 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_subcompactions: 1 2023-10-12T22:30:30.549 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_background_flushes: -1 2023-10-12T22:30:30.550 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-10-12T22:30:30.550 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-10-12T22:30:30.550 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-10-12T22:30:30.550 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-10-12T22:30:30.550 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-10-12T22:30:30.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.advise_random_on_open: 1 2023-10-12T22:30:30.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.db_write_buffer_size: 0 2023-10-12T22:30:30.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.write_buffer_manager: 0x561573695530 2023-10-12T22:30:30.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-10-12T22:30:30.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-10-12T22:30:30.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-10-12T22:30:30.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-10-12T22:30:30.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.rate_limiter: (nil) 2023-10-12T22:30:30.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-10-12T22:30:30.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.wal_recovery_mode: 2 2023-10-12T22:30:30.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.enable_thread_tracking: 0 2023-10-12T22:30:30.553 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.enable_pipelined_write: 0 2023-10-12T22:30:30.553 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.unordered_write: 0 2023-10-12T22:30:30.553 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-10-12T22:30:30.553 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-10-12T22:30:30.553 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-10-12T22:30:30.554 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-10-12T22:30:30.554 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.row_cache: None 2023-10-12T22:30:30.554 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.wal_filter: None 2023-10-12T22:30:30.554 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-10-12T22:30:30.554 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.allow_ingest_behind: 0 2023-10-12T22:30:30.555 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.preserve_deletes: 0 2023-10-12T22:30:30.555 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.two_write_queues: 0 2023-10-12T22:30:30.555 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.manual_wal_flush: 0 2023-10-12T22:30:30.555 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.atomic_flush: 0 2023-10-12T22:30:30.555 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-10-12T22:30:30.556 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-10-12T22:30:30.556 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-10-12T22:30:30.556 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.log_readahead_size: 0 2023-10-12T22:30:30.556 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-10-12T22:30:30.556 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_background_jobs: 2 2023-10-12T22:30:30.557 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_background_compactions: -1 2023-10-12T22:30:30.557 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-10-12T22:30:30.557 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-10-12T22:30:30.557 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-10-12T22:30:30.557 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_total_wal_size: 0 2023-10-12T22:30:30.558 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-10-12T22:30:30.558 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-10-12T22:30:30.558 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-10-12T22:30:30.558 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-10-12T22:30:30.558 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_open_files: -1 2023-10-12T22:30:30.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.bytes_per_sync: 0 2023-10-12T22:30:30.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-10-12T22:30:30.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-10-12T22:30:30.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_readahead_size: 0 2023-10-12T22:30:30.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Compression algorithms supported: 2023-10-12T22:30:30.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-10-12T22:30:30.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: kZSTD supported: 0 2023-10-12T22:30:30.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: kXpressCompression supported: 0 2023-10-12T22:30:30.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: kLZ4HCCompression supported: 1 2023-10-12T22:30:30.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: kLZ4Compression supported: 1 2023-10-12T22:30:30.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: kBZip2Compression supported: 0 2023-10-12T22:30:30.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: kZlibCompression supported: 1 2023-10-12T22:30:30.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: kSnappyCompression supported: 1 2023-10-12T22:30:30.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-10-12T22:30:30.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-10-12T22:30:30.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.562 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi131/store.db/MANIFEST-000001 2023-10-12T22:30:30.562 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.562 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-10-12T22:30:30.562 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.562 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-10-12T22:30:30.563 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.merge_operator: 2023-10-12T22:30:30.563 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_filter: None 2023-10-12T22:30:30.563 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_filter_factory: None 2023-10-12T22:30:30.563 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-10-12T22:30:30.563 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-10-12T22:30:30.563 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5615735a6120) 2023-10-12T22:30:30.564 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-10-12T22:30:30.564 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-10-12T22:30:30.564 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-10-12T22:30:30.564 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-10-12T22:30:30.564 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: index_type: 0 2023-10-12T22:30:30.564 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-10-12T22:30:30.565 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-10-12T22:30:30.565 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-10-12T22:30:30.565 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-10-12T22:30:30.565 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: checksum: 1 2023-10-12T22:30:30.565 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-10-12T22:30:30.566 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: block_cache: 0x5615735dcd10 2023-10-12T22:30:30.566 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-10-12T22:30:30.566 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-10-12T22:30:30.566 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-10-12T22:30:30.566 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-10-12T22:30:30.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-10-12T22:30:30.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-10-12T22:30:30.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-10-12T22:30:30.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-10-12T22:30:30.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-10-12T22:30:30.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-10-12T22:30:30.568 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-10-12T22:30:30.568 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-10-12T22:30:30.568 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-10-12T22:30:30.568 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-10-12T22:30:30.568 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-10-12T22:30:30.569 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-10-12T22:30:30.569 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-10-12T22:30:30.569 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-10-12T22:30:30.569 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-10-12T22:30:30.569 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: format_version: 2 2023-10-12T22:30:30.570 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-10-12T22:30:30.570 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: block_align: 0 2023-10-12T22:30:30.570 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.570 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.write_buffer_size: 33554432 2023-10-12T22:30:30.571 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_write_buffer_number: 2 2023-10-12T22:30:30.571 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compression: NoCompression 2023-10-12T22:30:30.571 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.bottommost_compression: Disabled 2023-10-12T22:30:30.571 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.prefix_extractor: nullptr 2023-10-12T22:30:30.571 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-10-12T22:30:30.572 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.num_levels: 7 2023-10-12T22:30:30.572 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-10-12T22:30:30.572 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-10-12T22:30:30.572 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-10-12T22:30:30.572 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-10-12T22:30:30.573 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-10-12T22:30:30.573 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-10-12T22:30:30.573 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-10-12T22:30:30.573 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-10-12T22:30:30.573 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-10-12T22:30:30.574 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-10-12T22:30:30.574 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compression_opts.level: 32767 2023-10-12T22:30:30.574 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compression_opts.strategy: 0 2023-10-12T22:30:30.574 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-10-12T22:30:30.574 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-10-12T22:30:30.575 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compression_opts.enabled: false 2023-10-12T22:30:30.575 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-10-12T22:30:30.575 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-10-12T22:30:30.575 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-10-12T22:30:30.575 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.target_file_size_base: 67108864 2023-10-12T22:30:30.575 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-10-12T22:30:30.576 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-10-12T22:30:30.576 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-10-12T22:30:30.576 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-10-12T22:30:30.576 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-10-12T22:30:30.576 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-10-12T22:30:30.576 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-10-12T22:30:30.577 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-10-12T22:30:30.577 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-10-12T22:30:30.577 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-10-12T22:30:30.577 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-10-12T22:30:30.577 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-10-12T22:30:30.577 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-10-12T22:30:30.578 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.arena_block_size: 4194304 2023-10-12T22:30:30.578 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-10-12T22:30:30.578 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-10-12T22:30:30.578 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-10-12T22:30:30.578 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.disable_auto_compactions: 0 2023-10-12T22:30:30.579 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-10-12T22:30:30.579 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-10-12T22:30:30.579 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-10-12T22:30:30.579 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-10-12T22:30:30.579 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-10-12T22:30:30.579 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-10-12T22:30:30.580 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-10-12T22:30:30.580 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-10-12T22:30:30.580 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-10-12T22:30:30.580 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-10-12T22:30:30.580 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.table_properties_collectors: 2023-10-12T22:30:30.581 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.inplace_update_support: 0 2023-10-12T22:30:30.581 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-10-12T22:30:30.581 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-10-12T22:30:30.581 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-10-12T22:30:30.581 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-10-12T22:30:30.582 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.bloom_locality: 0 2023-10-12T22:30:30.582 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.max_successive_merges: 0 2023-10-12T22:30:30.582 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-10-12T22:30:30.582 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.paranoid_file_checks: 0 2023-10-12T22:30:30.582 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.force_consistency_checks: 0 2023-10-12T22:30:30.583 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.report_bg_io_stats: 0 2023-10-12T22:30:30.583 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.ttl: 2592000 2023-10-12T22:30:30.583 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.516+0000 7fc114bba700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-10-12T22:30:30.583 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.517+0000 7fc114bba700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi131/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-10-12T22:30:30.583 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.584 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.517+0000 7fc114bba700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-10-12T22:30:30.584 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.584 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.518+0000 7fc114bba700 4 rocksdb: DB pointer 0x5615736a1800 2023-10-12T22:30:30.584 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.518+0000 7fc0fda53700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-10-12T22:30:30.584 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.518+0000 7fc0fda53700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-10-12T22:30:30.585 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-10-12T22:30:30.585 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-10-12T22:30:30.585 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-10-12T22:30:30.585 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-12T22:30:30.586 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-12T22:30:30.586 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-10-12T22:30:30.586 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-10-12T22:30:30.586 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-12T22:30:30.586 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.587 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-10-12T22:30:30.587 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-10-12T22:30:30.587 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-12T22:30:30.587 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-10-12T22:30:30.587 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-10-12T22:30:30.588 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.588 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-10-12T22:30:30.588 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-10-12T22:30:30.588 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-12T22:30:30.588 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-10-12T22:30:30.589 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-10-12T22:30:30.589 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-12T22:30:30.589 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-10-12T22:30:30.589 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-10-12T22:30:30.589 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-10-12T22:30:30.590 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-12T22:30:30.590 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-12T22:30:30.590 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-12T22:30:30.590 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.591 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-10-12T22:30:30.591 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.591 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-10-12T22:30:30.591 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-10-12T22:30:30.591 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-12T22:30:30.592 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-10-12T22:30:30.592 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-10-12T22:30:30.592 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.592 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-10-12T22:30:30.592 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-10-12T22:30:30.593 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-12T22:30:30.593 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-10-12T22:30:30.593 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-10-12T22:30:30.593 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-12T22:30:30.593 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-10-12T22:30:30.593 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-10-12T22:30:30.594 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-10-12T22:30:30.594 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-12T22:30:30.594 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-12T22:30:30.594 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-12T22:30:30.594 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.595 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-10-12T22:30:30.595 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: 2023-10-12T22:30:30.595 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.519+0000 7fc114bba700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-10-12T22:30:30.595 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.519+0000 7fc114bba700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-10-12T22:30:30.595 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-mon: debug 2023-10-12T22:30:30.519+0000 7fc114bba700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi131 for mon.smithi131 2023-10-12T22:30:30.866 INFO:teuthology.orchestra.run.smithi131.stderr:create mon.smithi131 on 2023-10-12T22:30:30.866 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:30:30.871 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl disable ceph.target 2023-10-12T22:30:30.878 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-10-12T22:30:30.966 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl enable ceph.target 2023-10-12T22:30:30.973 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-10-12T22:30:31.059 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl start ceph.target 2023-10-12T22:30:31.066 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl enable ceph-e4069eba-694e-11ee-8db6-212e2dc638e7.target 2023-10-12T22:30:31.072 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e4069eba-694e-11ee-8db6-212e2dc638e7.target → /etc/systemd/system/ceph-e4069eba-694e-11ee-8db6-212e2dc638e7.target. 2023-10-12T22:30:31.072 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-e4069eba-694e-11ee-8db6-212e2dc638e7.target → /etc/systemd/system/ceph-e4069eba-694e-11ee-8db6-212e2dc638e7.target. 2023-10-12T22:30:31.155 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl start ceph-e4069eba-694e-11ee-8db6-212e2dc638e7.target 2023-10-12T22:30:31.162 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl daemon-reload 2023-10-12T22:30:31.246 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl stop ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi131 2023-10-12T22:30:31.270 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl reset-failed ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi131 2023-10-12T22:30:31.275 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Failed to reset failed state of unit ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi131.service: Unit ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi131.service not loaded. 2023-10-12T22:30:31.276 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl enable ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi131 2023-10-12T22:30:31.282 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Created symlink /etc/systemd/system/ceph-e4069eba-694e-11ee-8db6-212e2dc638e7.target.wants/ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi131.service → /etc/systemd/system/ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@.service. 2023-10-12T22:30:31.368 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl start ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi131 2023-10-12T22:30:31.703 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-enabled firewalld.service 2023-10-12T22:30:31.713 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: disabled 2023-10-12T22:30:31.713 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-active firewalld.service 2023-10-12T22:30:31.720 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: inactive 2023-10-12T22:30:31.721 INFO:teuthology.orchestra.run.smithi131.stderr:firewalld.service is not enabled 2023-10-12T22:30:31.721 INFO:teuthology.orchestra.run.smithi131.stderr:Not possible to enable service . firewalld.service is not available 2023-10-12T22:30:31.721 INFO:teuthology.orchestra.run.smithi131.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-10-12T22:30:31.722 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for mon to start... 2023-10-12T22:30:31.722 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for mon... 2023-10-12T22:30:31.722 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/mon.smithi131:/var/lib/ceph/mon/ceph-smithi131:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-10-12T22:30:31.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:31 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112003]: cluster 2023-10-12T22:30:31.719920+0000 mon.smithi131 (mon.0) 1 : cluster [INF] mon.smithi131 is new leader, mons smithi131 in quorum (ranks 0) 2023-10-12T22:30:32.112 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.111+0000 7f846759e700 1 Processor -- start 2023-10-12T22:30:32.116 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.111+0000 7f846759e700 1 -- start start 2023-10-12T22:30:32.116 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.111+0000 7f846759e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 0x7f84680fbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:32.116 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.111+0000 7f846759e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84680fc2b0 con 0x7f84680fb9a0 2023-10-12T22:30:32.116 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.112+0000 7f846659c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 0x7f84680fbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:32.116 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.112+0000 7f846659c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 0x7f84680fbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48378/0 (socket says 172.21.15.131:48378) 2023-10-12T22:30:32.117 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.112+0000 7f846659c700 1 -- 172.21.15.131:0/3629773675 learned_addr learned my addr 172.21.15.131:0/3629773675 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:32.117 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.112+0000 7f846659c700 1 -- 172.21.15.131:0/3629773675 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84680fc3f0 con 0x7f84680fb9a0 2023-10-12T22:30:32.117 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.112+0000 7f846659c700 1 --2- 172.21.15.131:0/3629773675 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 0x7f84680fbdc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f845800b5f0 tx=0x7f8458006ee0).ready entity=mon.0 client_cookie=23702758a28e50bb server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:32.117 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.112+0000 7f846559a700 1 -- 172.21.15.131:0/3629773675 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f845800d600 con 0x7f84680fb9a0 2023-10-12T22:30:32.117 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.112+0000 7f846559a700 1 -- 172.21.15.131:0/3629773675 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845800d760 con 0x7f84680fb9a0 2023-10-12T22:30:32.118 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.112+0000 7f846559a700 1 -- 172.21.15.131:0/3629773675 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f845800da30 con 0x7f84680fb9a0 2023-10-12T22:30:32.118 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.113+0000 7f846759e700 1 -- 172.21.15.131:0/3629773675 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 msgr2=0x7f84680fbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:32.119 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.113+0000 7f846759e700 1 --2- 172.21.15.131:0/3629773675 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 0x7f84680fbdc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f845800b5f0 tx=0x7f8458006ee0).stop 2023-10-12T22:30:32.119 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.113+0000 7f846759e700 1 -- 172.21.15.131:0/3629773675 shutdown_connections 2023-10-12T22:30:32.119 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.113+0000 7f846759e700 1 --2- 172.21.15.131:0/3629773675 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 0x7f84680fbdc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:32.119 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.113+0000 7f846759e700 1 -- 172.21.15.131:0/3629773675 >> 172.21.15.131:0/3629773675 conn(0x7f84680f7200 msgr2=0x7f84680f9620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:32.120 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.113+0000 7f846759e700 1 -- 172.21.15.131:0/3629773675 shutdown_connections 2023-10-12T22:30:32.120 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.113+0000 7f846759e700 1 -- 172.21.15.131:0/3629773675 wait complete. 2023-10-12T22:30:32.120 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.114+0000 7f846759e700 1 Processor -- start 2023-10-12T22:30:32.120 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.114+0000 7f846759e700 1 -- start start 2023-10-12T22:30:32.120 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.114+0000 7f846759e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 0x7f8468109f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:32.121 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.114+0000 7f846759e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846810a480 con 0x7f84680fb9a0 2023-10-12T22:30:32.121 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.114+0000 7f846659c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 0x7f8468109f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:32.121 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.114+0000 7f846659c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 0x7f8468109f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48384/0 (socket says 172.21.15.131:48384) 2023-10-12T22:30:32.121 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.114+0000 7f846659c700 1 -- 172.21.15.131:0/270928603 learned_addr learned my addr 172.21.15.131:0/270928603 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:32.121 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.114+0000 7f846659c700 1 -- 172.21.15.131:0/270928603 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845800b200 con 0x7f84680fb9a0 2023-10-12T22:30:32.122 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.114+0000 7f846659c700 1 --2- 172.21.15.131:0/270928603 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 0x7f8468109f90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f845800c010 tx=0x7f8458016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:32.122 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.115+0000 7f8456ffd700 1 -- 172.21.15.131:0/270928603 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8458011cc0 con 0x7f84680fb9a0 2023-10-12T22:30:32.123 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.115+0000 7f8456ffd700 1 -- 172.21.15.131:0/270928603 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8458011e20 con 0x7f84680fb9a0 2023-10-12T22:30:32.123 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.115+0000 7f846759e700 1 -- 172.21.15.131:0/270928603 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846810a680 con 0x7f84680fb9a0 2023-10-12T22:30:32.123 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.115+0000 7f8456ffd700 1 -- 172.21.15.131:0/270928603 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f845801d590 con 0x7f84680fb9a0 2023-10-12T22:30:32.123 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.115+0000 7f846759e700 1 -- 172.21.15.131:0/270928603 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846810aa90 con 0x7f84680fb9a0 2023-10-12T22:30:32.123 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.115+0000 7f8456ffd700 1 -- 172.21.15.131:0/270928603 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f845800f070 con 0x7f84680fb9a0 2023-10-12T22:30:32.124 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.115+0000 7f8456ffd700 1 -- 172.21.15.131:0/270928603 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f84580277d0 con 0x7f84680fb9a0 2023-10-12T22:30:32.124 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.116+0000 7f846759e700 1 -- 172.21.15.131:0/270928603 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8468045ad0 con 0x7f84680fb9a0 2023-10-12T22:30:32.124 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.118+0000 7f8456ffd700 1 -- 172.21.15.131:0/270928603 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8458016070 con 0x7f84680fb9a0 2023-10-12T22:30:32.187 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.183+0000 7f846759e700 1 -- 172.21.15.131:0/270928603 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f84681039f0 con 0x7f84680fb9a0 2023-10-12T22:30:32.187 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.184+0000 7f8456ffd700 1 -- 172.21.15.131:0/270928603 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f8458036070 con 0x7f84680fb9a0 2023-10-12T22:30:32.187 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: cluster: 2023-10-12T22:30:32.188 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: id: e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:30:32.188 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-10-12T22:30:32.188 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.188 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: services: 2023-10-12T22:30:32.188 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi131 (age 0.449604s) 2023-10-12T22:30:32.189 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mgr: no daemons active 2023-10-12T22:30:32.189 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-10-12T22:30:32.189 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.189 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: data: 2023-10-12T22:30:32.189 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-10-12T22:30:32.190 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-10-12T22:30:32.190 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-10-12T22:30:32.190 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: pgs: 2023-10-12T22:30:32.190 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.190 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.184+0000 7f846759e700 1 -- 172.21.15.131:0/270928603 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 msgr2=0x7f8468109f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:32.191 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.184+0000 7f846759e700 1 --2- 172.21.15.131:0/270928603 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 0x7f8468109f90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f845800c010 tx=0x7f8458016000).stop 2023-10-12T22:30:32.191 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.184+0000 7f846759e700 1 -- 172.21.15.131:0/270928603 shutdown_connections 2023-10-12T22:30:32.191 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.184+0000 7f846759e700 1 --2- 172.21.15.131:0/270928603 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84680fb9a0 0x7f8468109f90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:32.191 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.184+0000 7f846759e700 1 -- 172.21.15.131:0/270928603 >> 172.21.15.131:0/270928603 conn(0x7f84680f7200 msgr2=0x7f84680fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:32.191 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.184+0000 7f846759e700 1 -- 172.21.15.131:0/270928603 shutdown_connections 2023-10-12T22:30:32.192 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.185+0000 7f846759e700 1 -- 172.21.15.131:0/270928603 wait complete. 2023-10-12T22:30:32.527 INFO:teuthology.orchestra.run.smithi131.stderr:mon is available 2023-10-12T22:30:32.528 INFO:teuthology.orchestra.run.smithi131.stderr:Assimilating anything we can from ceph.conf... 2023-10-12T22:30:32.528 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z -v /var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/mon.smithi131:/var/lib/ceph/mon/ceph-smithi131:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi131/config 2023-10-12T22:30:32.925 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.921+0000 7f6cb21a0700 1 Processor -- start 2023-10-12T22:30:32.925 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.922+0000 7f6cb21a0700 1 -- start start 2023-10-12T22:30:32.925 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.922+0000 7f6cb21a0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 0x7f6cac0fe650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:32.925 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.922+0000 7f6cb21a0700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac0feb40 con 0x7f6cac06fc30 2023-10-12T22:30:32.926 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.922+0000 7f6cb119e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 0x7f6cac0fe650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:32.926 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.922+0000 7f6cb119e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 0x7f6cac0fe650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48394/0 (socket says 172.21.15.131:48394) 2023-10-12T22:30:32.926 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.922+0000 7f6cb119e700 1 -- 172.21.15.131:0/3074330697 learned_addr learned my addr 172.21.15.131:0/3074330697 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:32.926 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.922+0000 7f6cb119e700 1 -- 172.21.15.131:0/3074330697 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cac0fec80 con 0x7f6cac06fc30 2023-10-12T22:30:32.926 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.923+0000 7f6cb119e700 1 --2- 172.21.15.131:0/3074330697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 0x7f6cac0fe650 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f6ca000b5f0 tx=0x7f6ca0006ee0).ready entity=mon.0 client_cookie=761a2fe00a133823 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:32.927 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.923+0000 7f6c9ffff700 1 -- 172.21.15.131:0/3074330697 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ca000d600 con 0x7f6cac06fc30 2023-10-12T22:30:32.927 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.923+0000 7f6c9ffff700 1 -- 172.21.15.131:0/3074330697 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca000d760 con 0x7f6cac06fc30 2023-10-12T22:30:32.927 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.923+0000 7f6c9ffff700 1 -- 172.21.15.131:0/3074330697 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ca000da30 con 0x7f6cac06fc30 2023-10-12T22:30:32.927 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.923+0000 7f6cb21a0700 1 -- 172.21.15.131:0/3074330697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 msgr2=0x7f6cac0fe650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:32.927 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.923+0000 7f6cb21a0700 1 --2- 172.21.15.131:0/3074330697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 0x7f6cac0fe650 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f6ca000b5f0 tx=0x7f6ca0006ee0).stop 2023-10-12T22:30:32.928 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.923+0000 7f6cb21a0700 1 -- 172.21.15.131:0/3074330697 shutdown_connections 2023-10-12T22:30:32.928 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.923+0000 7f6cb21a0700 1 --2- 172.21.15.131:0/3074330697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 0x7f6cac0fe650 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:32.928 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.923+0000 7f6cb21a0700 1 -- 172.21.15.131:0/3074330697 >> 172.21.15.131:0/3074330697 conn(0x7f6cac0f6130 msgr2=0x7f6cac0f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:32.928 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.923+0000 7f6cb21a0700 1 -- 172.21.15.131:0/3074330697 shutdown_connections 2023-10-12T22:30:32.928 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.923+0000 7f6cb21a0700 1 -- 172.21.15.131:0/3074330697 wait complete. 2023-10-12T22:30:32.930 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.924+0000 7f6cb21a0700 1 Processor -- start 2023-10-12T22:30:32.931 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.924+0000 7f6cb21a0700 1 -- start start 2023-10-12T22:30:32.931 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.924+0000 7f6cb21a0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 0x7f6cac10a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:32.931 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.924+0000 7f6cb21a0700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac10a500 con 0x7f6cac06fc30 2023-10-12T22:30:32.931 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.925+0000 7f6cb119e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 0x7f6cac10a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:32.931 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.925+0000 7f6cb119e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 0x7f6cac10a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48408/0 (socket says 172.21.15.131:48408) 2023-10-12T22:30:32.932 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.925+0000 7f6cb119e700 1 -- 172.21.15.131:0/1232548477 learned_addr learned my addr 172.21.15.131:0/1232548477 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:32.932 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.925+0000 7f6cb119e700 1 -- 172.21.15.131:0/1232548477 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca000b200 con 0x7f6cac06fc30 2023-10-12T22:30:32.932 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.925+0000 7f6cb119e700 1 --2- 172.21.15.131:0/1232548477 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 0x7f6cac10a010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f6ca0006c70 tx=0x7f6ca0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:32.932 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.925+0000 7f6c9dffb700 1 -- 172.21.15.131:0/1232548477 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ca0011b40 con 0x7f6cac06fc30 2023-10-12T22:30:32.932 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.925+0000 7f6c9dffb700 1 -- 172.21.15.131:0/1232548477 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca0011ca0 con 0x7f6cac06fc30 2023-10-12T22:30:32.933 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.925+0000 7f6c9dffb700 1 -- 172.21.15.131:0/1232548477 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ca001d430 con 0x7f6cac06fc30 2023-10-12T22:30:32.933 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.925+0000 7f6cb21a0700 1 -- 172.21.15.131:0/1232548477 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cac10a700 con 0x7f6cac06fc30 2023-10-12T22:30:32.933 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.925+0000 7f6cb21a0700 1 -- 172.21.15.131:0/1232548477 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cac10ab10 con 0x7f6cac06fc30 2023-10-12T22:30:32.933 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.926+0000 7f6c9dffb700 1 -- 172.21.15.131:0/1232548477 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f6ca000f070 con 0x7f6cac06fc30 2023-10-12T22:30:32.933 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.926+0000 7f6c9dffb700 1 -- 172.21.15.131:0/1232548477 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6ca0027460 con 0x7f6cac06fc30 2023-10-12T22:30:32.934 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.927+0000 7f6cb21a0700 1 -- 172.21.15.131:0/1232548477 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cac045ad0 con 0x7f6cac06fc30 2023-10-12T22:30:32.934 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.929+0000 7f6c9dffb700 1 -- 172.21.15.131:0/1232548477 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6ca0016070 con 0x7f6cac06fc30 2023-10-12T22:30:32.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112003]: cluster 2023-10-12T22:30:31.735603+0000 mon.smithi131 (mon.0) 2 : cluster [INF] mon.smithi131 is new leader, mons smithi131 in quorum (ranks 0) 2023-10-12T22:30:32.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112003]: cluster 2023-10-12T22:30:31.735849+0000 mon.smithi131 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi131=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} 2023-10-12T22:30:32.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112003]: cluster 2023-10-12T22:30:31.738580+0000 mon.smithi131 (mon.0) 4 : cluster [DBG] fsmap 2023-10-12T22:30:32.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112003]: cluster 2023-10-12T22:30:31.741819+0000 mon.smithi131 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-10-12T22:30:32.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112003]: cluster 2023-10-12T22:30:31.742285+0000 mon.smithi131 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-10-12T22:30:32.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112003]: audit 2023-10-12T22:30:32.185173+0000 mon.smithi131 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.131:0/270928603' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-10-12T22:30:32.994 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.993+0000 7f6cb21a0700 1 -- 172.21.15.131:0/1232548477 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6cac0fef30 con 0x7f6cac06fc30 2023-10-12T22:30:33.000 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.997+0000 7f6c9dffb700 1 -- 172.21.15.131:0/1232548477 <== mon.0 v2:172.21.15.131:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6ca001d660 con 0x7f6cac06fc30 2023-10-12T22:30:33.000 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.997+0000 7f6c9dffb700 1 -- 172.21.15.131:0/1232548477 <== mon.0 v2:172.21.15.131:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+466 (secure 0 0 0) 0x7f6ca0036020 con 0x7f6cac06fc30 2023-10-12T22:30:33.001 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.001 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: [global] 2023-10-12T22:30:33.001 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: fsid = e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:30:33.001 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.131:3300,v1:172.21.15.131:6789] 2023-10-12T22:30:33.001 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-10-12T22:30:33.002 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-10-12T22:30:33.002 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-10-12T22:30:33.002 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-10-12T22:30:33.002 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.002 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: [mgr] 2023-10-12T22:30:33.003 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-10-12T22:30:33.003 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.003 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: [osd] 2023-10-12T22:30:33.003 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-10-12T22:30:33.003 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-10-12T22:30:33.003 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-10-12T22:30:33.004 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-10-12T22:30:33.004 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-10-12T22:30:33.004 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-10-12T22:30:33.004 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.998+0000 7f6cb21a0700 1 -- 172.21.15.131:0/1232548477 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 msgr2=0x7f6cac10a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:33.004 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.998+0000 7f6cb21a0700 1 --2- 172.21.15.131:0/1232548477 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 0x7f6cac10a010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f6ca0006c70 tx=0x7f6ca0016000).stop 2023-10-12T22:30:33.005 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.998+0000 7f6cb21a0700 1 -- 172.21.15.131:0/1232548477 shutdown_connections 2023-10-12T22:30:33.005 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.998+0000 7f6cb21a0700 1 --2- 172.21.15.131:0/1232548477 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cac06fc30 0x7f6cac10a010 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:33.005 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.998+0000 7f6cb21a0700 1 -- 172.21.15.131:0/1232548477 >> 172.21.15.131:0/1232548477 conn(0x7f6cac0f6130 msgr2=0x7f6cac0f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:33.005 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.998+0000 7f6cb21a0700 1 -- 172.21.15.131:0/1232548477 shutdown_connections 2023-10-12T22:30:33.005 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:32.998+0000 7f6cb21a0700 1 -- 172.21.15.131:0/1232548477 wait complete. 2023-10-12T22:30:33.335 INFO:teuthology.orchestra.run.smithi131.stderr:Generating new minimal ceph.conf... 2023-10-12T22:30:33.336 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z -v /var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/mon.smithi131:/var/lib/ceph/mon/ceph-smithi131:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi131/config 2023-10-12T22:30:33.688 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.684+0000 7fbcba27a700 1 Processor -- start 2023-10-12T22:30:33.688 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.684+0000 7fbcba27a700 1 -- start start 2023-10-12T22:30:33.688 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.684+0000 7fbcba27a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 0x7fbcb4100be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:33.688 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.684+0000 7fbcba27a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb40fc660 con 0x7fbcb406fc90 2023-10-12T22:30:33.689 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.685+0000 7fbcb3fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 0x7fbcb4100be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:33.689 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.685+0000 7fbcb3fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 0x7fbcb4100be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48416/0 (socket says 172.21.15.131:48416) 2023-10-12T22:30:33.689 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.685+0000 7fbcb3fff700 1 -- 172.21.15.131:0/2111642915 learned_addr learned my addr 172.21.15.131:0/2111642915 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:33.689 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.685+0000 7fbcb3fff700 1 -- 172.21.15.131:0/2111642915 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb41010d0 con 0x7fbcb406fc90 2023-10-12T22:30:33.689 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.685+0000 7fbcb3fff700 1 --2- 172.21.15.131:0/2111642915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 0x7fbcb4100be0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fbca400b5f0 tx=0x7fbca4006ee0).ready entity=mon.0 client_cookie=b6cbe2d3ac04d5fa server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:33.690 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.685+0000 7fbcb2ffd700 1 -- 172.21.15.131:0/2111642915 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbca400d600 con 0x7fbcb406fc90 2023-10-12T22:30:33.690 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.685+0000 7fbcb2ffd700 1 -- 172.21.15.131:0/2111642915 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbca400dbd0 con 0x7fbcb406fc90 2023-10-12T22:30:33.690 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.685+0000 7fbcb2ffd700 1 -- 172.21.15.131:0/2111642915 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbca4011c00 con 0x7fbcb406fc90 2023-10-12T22:30:33.690 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.686+0000 7fbcba27a700 1 -- 172.21.15.131:0/2111642915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 msgr2=0x7fbcb4100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:33.690 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.686+0000 7fbcba27a700 1 --2- 172.21.15.131:0/2111642915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 0x7fbcb4100be0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fbca400b5f0 tx=0x7fbca4006ee0).stop 2023-10-12T22:30:33.691 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.686+0000 7fbcba27a700 1 -- 172.21.15.131:0/2111642915 shutdown_connections 2023-10-12T22:30:33.691 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.686+0000 7fbcba27a700 1 --2- 172.21.15.131:0/2111642915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 0x7fbcb4100be0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:33.691 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.686+0000 7fbcba27a700 1 -- 172.21.15.131:0/2111642915 >> 172.21.15.131:0/2111642915 conn(0x7fbcb40f6280 msgr2=0x7fbcb40f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:33.691 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.686+0000 7fbcba27a700 1 -- 172.21.15.131:0/2111642915 shutdown_connections 2023-10-12T22:30:33.691 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.686+0000 7fbcba27a700 1 -- 172.21.15.131:0/2111642915 wait complete. 2023-10-12T22:30:33.691 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.686+0000 7fbcba27a700 1 Processor -- start 2023-10-12T22:30:33.692 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.687+0000 7fbcba27a700 1 -- start start 2023-10-12T22:30:33.693 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.687+0000 7fbcba27a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 0x7fbcb410cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:33.693 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.687+0000 7fbcba27a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb410d140 con 0x7fbcb406fc90 2023-10-12T22:30:33.693 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.687+0000 7fbcb3fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 0x7fbcb410cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:33.694 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.687+0000 7fbcb3fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 0x7fbcb410cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48432/0 (socket says 172.21.15.131:48432) 2023-10-12T22:30:33.694 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.687+0000 7fbcb3fff700 1 -- 172.21.15.131:0/3421759528 learned_addr learned my addr 172.21.15.131:0/3421759528 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:33.694 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.687+0000 7fbcb3fff700 1 -- 172.21.15.131:0/3421759528 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca400b200 con 0x7fbcb406fc90 2023-10-12T22:30:33.694 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.687+0000 7fbcb3fff700 1 --2- 172.21.15.131:0/3421759528 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 0x7fbcb410cc50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbca4000c00 tx=0x7fbca4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:33.694 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.688+0000 7fbcb0ff9700 1 -- 172.21.15.131:0/3421759528 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbca4011690 con 0x7fbcb406fc90 2023-10-12T22:30:33.695 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.688+0000 7fbcb0ff9700 1 -- 172.21.15.131:0/3421759528 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbca400d9f0 con 0x7fbcb406fc90 2023-10-12T22:30:33.695 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.688+0000 7fbcba27a700 1 -- 172.21.15.131:0/3421759528 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb410d340 con 0x7fbcb406fc90 2023-10-12T22:30:33.695 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.688+0000 7fbcb0ff9700 1 -- 172.21.15.131:0/3421759528 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbca401b710 con 0x7fbcb406fc90 2023-10-12T22:30:33.696 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.688+0000 7fbcba27a700 1 -- 172.21.15.131:0/3421759528 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb410d750 con 0x7fbcb406fc90 2023-10-12T22:30:33.696 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.688+0000 7fbcb0ff9700 1 -- 172.21.15.131:0/3421759528 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fbca400f070 con 0x7fbcb406fc90 2023-10-12T22:30:33.696 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.688+0000 7fbcb0ff9700 1 -- 172.21.15.131:0/3421759528 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fbca4025a30 con 0x7fbcb406fc90 2023-10-12T22:30:33.696 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.689+0000 7fbcba27a700 1 -- 172.21.15.131:0/3421759528 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb4045ad0 con 0x7fbcb406fc90 2023-10-12T22:30:33.696 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.691+0000 7fbcb0ff9700 1 -- 172.21.15.131:0/3421759528 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fbca4016070 con 0x7fbcb406fc90 2023-10-12T22:30:33.758 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.757+0000 7fbcba27a700 1 -- 172.21.15.131:0/3421759528 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fbcb4101e00 con 0x7fbcb406fc90 2023-10-12T22:30:33.759 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.757+0000 7fbcb0ff9700 1 -- 172.21.15.131:0/3421759528 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fbca401aaf0 con 0x7fbcb406fc90 2023-10-12T22:30:33.760 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.758+0000 7fbcba27a700 1 -- 172.21.15.131:0/3421759528 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 msgr2=0x7fbcb410cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:33.760 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.758+0000 7fbcba27a700 1 --2- 172.21.15.131:0/3421759528 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 0x7fbcb410cc50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbca4000c00 tx=0x7fbca4006ee0).stop 2023-10-12T22:30:33.760 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.758+0000 7fbcba27a700 1 -- 172.21.15.131:0/3421759528 shutdown_connections 2023-10-12T22:30:33.760 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.758+0000 7fbcba27a700 1 --2- 172.21.15.131:0/3421759528 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbcb406fc90 0x7fbcb410cc50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:33.761 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.758+0000 7fbcba27a700 1 -- 172.21.15.131:0/3421759528 >> 172.21.15.131:0/3421759528 conn(0x7fbcb40f6280 msgr2=0x7fbcb40f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:33.761 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.758+0000 7fbcba27a700 1 -- 172.21.15.131:0/3421759528 shutdown_connections 2023-10-12T22:30:33.761 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:33.758+0000 7fbcba27a700 1 -- 172.21.15.131:0/3421759528 wait complete. 2023-10-12T22:30:34.012 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:33 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112003]: audit 2023-10-12T22:30:32.995111+0000 mon.smithi131 (mon.0) 8 : audit [INF] from='client.? 172.21.15.131:0/1232548477' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-10-12T22:30:34.013 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:33 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112003]: audit 2023-10-12T22:30:32.998281+0000 mon.smithi131 (mon.0) 9 : audit [INF] from='client.? 172.21.15.131:0/1232548477' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-10-12T22:30:34.102 INFO:teuthology.orchestra.run.smithi131.stderr:Restarting the monitor... 2023-10-12T22:30:34.103 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl restart ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi131 2023-10-12T22:30:34.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 systemd[1]: Stopping Ceph mon.smithi131 for e4069eba-694e-11ee-8db6-212e2dc638e7... 2023-10-12T22:30:34.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112003]: debug 2023-10-12T22:30:34.182+0000 7f312f3c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi131 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-10-12T22:30:34.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112003]: debug 2023-10-12T22:30:34.182+0000 7f312f3c5700 -1 mon.smithi131@0(leader) e1 *** Got Signal Terminated *** 2023-10-12T22:30:34.860 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 podman[112325]: ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131 2023-10-12T22:30:34.860 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 systemd[1]: ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi131.service: Succeeded. 2023-10-12T22:30:34.861 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 systemd[1]: Stopped Ceph mon.smithi131 for e4069eba-694e-11ee-8db6-212e2dc638e7. 2023-10-12T22:30:34.861 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 systemd[1]: Starting Ceph mon.smithi131 for e4069eba-694e-11ee-8db6-212e2dc638e7... 2023-10-12T22:30:34.910 INFO:teuthology.orchestra.run.smithi131.stderr:Setting mon public_network to 172.21.0.0/20 2023-10-12T22:30:34.911 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-10-12T22:30:35.241 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.905+0000 7f7ee0eea700 0 set uid:gid to 167:167 (ceph:ceph) 2023-10-12T22:30:35.241 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.905+0000 7f7ee0eea700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-10-12T22:30:35.242 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 bash[112396]: a101a867f1b1af1a0757559f84c46b36fb457e0b52ab5e8261e370c1823c0605 2023-10-12T22:30:35.242 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.905+0000 7f7ee0eea700 0 pidfile_write: ignore empty --pid-file 2023-10-12T22:30:35.242 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 systemd[1]: Started Ceph mon.smithi131 for e4069eba-694e-11ee-8db6-212e2dc638e7. 2023-10-12T22:30:35.242 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.910+0000 7f7ee0eea700 0 load: jerasure load: lrc load: isa 2023-10-12T22:30:35.242 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: RocksDB version: 6.8.1 2023-10-12T22:30:35.243 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.243 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-10-12T22:30:35.243 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Compile date May 13 2021 2023-10-12T22:30:35.243 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: DB SUMMARY 2023-10-12T22:30:35.243 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.244 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: CURRENT file: CURRENT 2023-10-12T22:30:35.244 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.244 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: IDENTITY file: IDENTITY 2023-10-12T22:30:35.244 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.244 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-10-12T22:30:35.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi131/store.db dir, Total Num: 1, files: 000004.sst 2023-10-12T22:30:35.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi131/store.db: 000006.log size: 92470 ; 2023-10-12T22:30:35.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.246 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.error_if_exists: 0 2023-10-12T22:30:35.246 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.create_if_missing: 0 2023-10-12T22:30:35.246 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.paranoid_checks: 1 2023-10-12T22:30:35.246 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.env: 0x564ca27641c0 2023-10-12T22:30:35.246 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.fs: Posix File System 2023-10-12T22:30:35.247 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.info_log: 0x564ca3819960 2023-10-12T22:30:35.247 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_file_opening_threads: 16 2023-10-12T22:30:35.247 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.statistics: (nil) 2023-10-12T22:30:35.247 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.use_fsync: 0 2023-10-12T22:30:35.247 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_log_file_size: 0 2023-10-12T22:30:35.248 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-10-12T22:30:35.248 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-10-12T22:30:35.248 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.keep_log_file_num: 1000 2023-10-12T22:30:35.248 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.recycle_log_file_num: 0 2023-10-12T22:30:35.248 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.allow_fallocate: 1 2023-10-12T22:30:35.248 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.allow_mmap_reads: 0 2023-10-12T22:30:35.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.allow_mmap_writes: 0 2023-10-12T22:30:35.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.use_direct_reads: 0 2023-10-12T22:30:35.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-10-12T22:30:35.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.create_missing_column_families: 0 2023-10-12T22:30:35.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.db_log_dir: 2023-10-12T22:30:35.250 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi131/store.db 2023-10-12T22:30:35.250 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-10-12T22:30:35.250 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_subcompactions: 1 2023-10-12T22:30:35.250 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_background_flushes: -1 2023-10-12T22:30:35.250 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-10-12T22:30:35.251 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-10-12T22:30:35.251 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-10-12T22:30:35.251 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-10-12T22:30:35.251 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-10-12T22:30:35.251 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.advise_random_on_open: 1 2023-10-12T22:30:35.252 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.db_write_buffer_size: 0 2023-10-12T22:30:35.252 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.write_buffer_manager: 0x564ca381ffb0 2023-10-12T22:30:35.252 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-10-12T22:30:35.252 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-10-12T22:30:35.252 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-10-12T22:30:35.253 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-10-12T22:30:35.253 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.rate_limiter: (nil) 2023-10-12T22:30:35.253 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-10-12T22:30:35.253 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.wal_recovery_mode: 2 2023-10-12T22:30:35.253 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.enable_thread_tracking: 0 2023-10-12T22:30:35.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.enable_pipelined_write: 0 2023-10-12T22:30:35.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.unordered_write: 0 2023-10-12T22:30:35.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-10-12T22:30:35.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-10-12T22:30:35.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-10-12T22:30:35.255 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-10-12T22:30:35.255 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.row_cache: None 2023-10-12T22:30:35.255 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.wal_filter: None 2023-10-12T22:30:35.255 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-10-12T22:30:35.255 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.allow_ingest_behind: 0 2023-10-12T22:30:35.255 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.preserve_deletes: 0 2023-10-12T22:30:35.256 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.two_write_queues: 0 2023-10-12T22:30:35.256 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.manual_wal_flush: 0 2023-10-12T22:30:35.256 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.atomic_flush: 0 2023-10-12T22:30:35.256 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-10-12T22:30:35.256 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-10-12T22:30:35.257 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-10-12T22:30:35.257 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.log_readahead_size: 0 2023-10-12T22:30:35.257 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-10-12T22:30:35.257 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_background_jobs: 2 2023-10-12T22:30:35.257 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_background_compactions: -1 2023-10-12T22:30:35.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-10-12T22:30:35.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-10-12T22:30:35.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-10-12T22:30:35.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_total_wal_size: 0 2023-10-12T22:30:35.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-10-12T22:30:35.259 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-10-12T22:30:35.259 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-10-12T22:30:35.259 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-10-12T22:30:35.259 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_open_files: -1 2023-10-12T22:30:35.259 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.bytes_per_sync: 0 2023-10-12T22:30:35.259 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-10-12T22:30:35.260 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-10-12T22:30:35.260 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_readahead_size: 0 2023-10-12T22:30:35.260 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Compression algorithms supported: 2023-10-12T22:30:35.260 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-10-12T22:30:35.260 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: kZSTD supported: 0 2023-10-12T22:30:35.261 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: kXpressCompression supported: 0 2023-10-12T22:30:35.261 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: kLZ4HCCompression supported: 1 2023-10-12T22:30:35.261 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: kLZ4Compression supported: 1 2023-10-12T22:30:35.261 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: kBZip2Compression supported: 0 2023-10-12T22:30:35.261 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: kZlibCompression supported: 1 2023-10-12T22:30:35.261 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: kSnappyCompression supported: 1 2023-10-12T22:30:35.262 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-10-12T22:30:35.262 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi131/store.db/MANIFEST-000005 2023-10-12T22:30:35.262 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.262 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-10-12T22:30:35.262 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.263 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-10-12T22:30:35.263 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.merge_operator: 2023-10-12T22:30:35.263 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_filter: None 2023-10-12T22:30:35.263 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_filter_factory: None 2023-10-12T22:30:35.263 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-10-12T22:30:35.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-10-12T22:30:35.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564ca3730128) 2023-10-12T22:30:35.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cache_index_and_filter_blocks: 1 2023-10-12T22:30:35.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cache_index_and_filter_blocks_with_high_priority: 0 2023-10-12T22:30:35.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-10-12T22:30:35.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: pin_top_level_index_and_filter: 1 2023-10-12T22:30:35.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: index_type: 0 2023-10-12T22:30:35.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: data_block_index_type: 0 2023-10-12T22:30:35.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: index_shortening: 1 2023-10-12T22:30:35.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: data_block_hash_table_util_ratio: 0.750000 2023-10-12T22:30:35.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: hash_index_allow_collision: 1 2023-10-12T22:30:35.266 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: checksum: 1 2023-10-12T22:30:35.266 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: no_block_cache: 0 2023-10-12T22:30:35.266 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: block_cache: 0x564ca3766f10 2023-10-12T22:30:35.266 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: block_cache_name: BinnedLRUCache 2023-10-12T22:30:35.266 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: block_cache_options: 2023-10-12T22:30:35.267 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: capacity : 536870912 2023-10-12T22:30:35.268 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: num_shard_bits : 4 2023-10-12T22:30:35.269 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: strict_capacity_limit : 0 2023-10-12T22:30:35.269 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: high_pri_pool_ratio: 0.000 2023-10-12T22:30:35.269 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: block_cache_compressed: (nil) 2023-10-12T22:30:35.269 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: persistent_cache: (nil) 2023-10-12T22:30:35.269 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: block_size: 4096 2023-10-12T22:30:35.269 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: block_size_deviation: 10 2023-10-12T22:30:35.270 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: block_restart_interval: 16 2023-10-12T22:30:35.270 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: index_block_restart_interval: 1 2023-10-12T22:30:35.270 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: metadata_block_size: 4096 2023-10-12T22:30:35.270 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: partition_filters: 0 2023-10-12T22:30:35.270 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: use_delta_encoding: 1 2023-10-12T22:30:35.271 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: filter_policy: rocksdb.BuiltinBloomFilter 2023-10-12T22:30:35.271 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: whole_key_filtering: 1 2023-10-12T22:30:35.271 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: verify_compression: 0 2023-10-12T22:30:35.271 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: read_amp_bytes_per_bit: 0 2023-10-12T22:30:35.271 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: format_version: 2 2023-10-12T22:30:35.272 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: enable_index_compression: 1 2023-10-12T22:30:35.272 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: block_align: 0 2023-10-12T22:30:35.272 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.272 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.write_buffer_size: 33554432 2023-10-12T22:30:35.272 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_write_buffer_number: 2 2023-10-12T22:30:35.272 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compression: NoCompression 2023-10-12T22:30:35.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.bottommost_compression: Disabled 2023-10-12T22:30:35.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.prefix_extractor: nullptr 2023-10-12T22:30:35.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-10-12T22:30:35.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.num_levels: 7 2023-10-12T22:30:35.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-10-12T22:30:35.274 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-10-12T22:30:35.274 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-10-12T22:30:35.274 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-10-12T22:30:35.274 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-10-12T22:30:35.274 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-10-12T22:30:35.275 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-10-12T22:30:35.275 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-10-12T22:30:35.275 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-10-12T22:30:35.275 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-10-12T22:30:35.275 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compression_opts.level: 32767 2023-10-12T22:30:35.276 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compression_opts.strategy: 0 2023-10-12T22:30:35.276 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-10-12T22:30:35.276 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-10-12T22:30:35.276 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compression_opts.enabled: false 2023-10-12T22:30:35.276 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-10-12T22:30:35.277 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-10-12T22:30:35.277 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-10-12T22:30:35.277 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.target_file_size_base: 67108864 2023-10-12T22:30:35.277 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-10-12T22:30:35.277 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-10-12T22:30:35.278 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-10-12T22:30:35.278 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-10-12T22:30:35.278 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-10-12T22:30:35.278 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-10-12T22:30:35.278 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-10-12T22:30:35.279 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-10-12T22:30:35.279 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-10-12T22:30:35.279 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-10-12T22:30:35.279 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-10-12T22:30:35.279 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-10-12T22:30:35.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-10-12T22:30:35.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.arena_block_size: 4194304 2023-10-12T22:30:35.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-10-12T22:30:35.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-10-12T22:30:35.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-10-12T22:30:35.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.disable_auto_compactions: 0 2023-10-12T22:30:35.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-10-12T22:30:35.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-10-12T22:30:35.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-10-12T22:30:35.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-10-12T22:30:35.282 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-10-12T22:30:35.282 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-10-12T22:30:35.282 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-10-12T22:30:35.282 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-10-12T22:30:35.282 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-10-12T22:30:35.283 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-10-12T22:30:35.283 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.table_properties_collectors: 2023-10-12T22:30:35.283 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.inplace_update_support: 0 2023-10-12T22:30:35.283 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-10-12T22:30:35.283 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-10-12T22:30:35.284 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-10-12T22:30:35.284 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-10-12T22:30:35.284 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.bloom_locality: 0 2023-10-12T22:30:35.284 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.max_successive_merges: 0 2023-10-12T22:30:35.284 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-10-12T22:30:35.284 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.paranoid_file_checks: 0 2023-10-12T22:30:35.285 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.force_consistency_checks: 0 2023-10-12T22:30:35.285 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.report_bg_io_stats: 0 2023-10-12T22:30:35.285 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.ttl: 2592000 2023-10-12T22:30:35.285 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-10-12T22:30:35.285 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi131/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-10-12T22:30:35.286 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.286 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-10-12T22:30:35.286 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.286 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1697149834913108, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-10-12T22:30:35.286 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.911+0000 7f7ee0eea700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-10-12T22:30:35.287 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.912+0000 7f7ee0eea700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-10-12T22:30:35.287 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.912+0000 7f7ee0eea700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1697149834913969, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88270, "table_properties": {"data_size": 86397, "index_size": 350, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12269, "raw_average_key_size": 49, "raw_value_size": 79611, "raw_average_value_size": 322, "num_data_blocks": 11, "num_entries": 247, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1697149834, "oldest_key_time": 3, "file_creation_time": 0}} 2023-10-12T22:30:35.287 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.912+0000 7f7ee0eea700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-10-12T22:30:35.287 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.287 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.913+0000 7f7ee0eea700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1697149834914371, "job": 1, "event": "recovery_finished"} 2023-10-12T22:30:35.288 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.913+0000 7f7ee0eea700 4 rocksdb: DB pointer 0x564ca382b800 2023-10-12T22:30:35.288 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.914+0000 7f7ec7d02700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-10-12T22:30:35.288 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.914+0000 7f7ec7d02700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-10-12T22:30:35.288 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ** DB Stats ** 2023-10-12T22:30:35.288 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Uptime(secs): 0.0 total, 0.0 interval 2023-10-12T22:30:35.288 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-10-12T22:30:35.289 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-12T22:30:35.289 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-12T22:30:35.289 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-10-12T22:30:35.289 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-10-12T22:30:35.289 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-12T22:30:35.290 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.290 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ** Compaction Stats [default] ** 2023-10-12T22:30:35.290 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-10-12T22:30:35.290 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-12T22:30:35.290 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: L0 2/0 87.89 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.5 0.00 0.00 1 0.001 0 0 2023-10-12T22:30:35.291 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Sum 2/0 87.89 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.5 0.00 0.00 1 0.001 0 0 2023-10-12T22:30:35.291 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.5 0.00 0.00 1 0.001 0 0 2023-10-12T22:30:35.291 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.291 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ** Compaction Stats [default] ** 2023-10-12T22:30:35.291 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-10-12T22:30:35.291 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-12T22:30:35.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.5 0.00 0.00 1 0.001 0 0 2023-10-12T22:30:35.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Uptime(secs): 0.0 total, 0.0 interval 2023-10-12T22:30:35.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Flush(GB): cumulative 0.000, interval 0.000 2023-10-12T22:30:35.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-12T22:30:35.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: AddFile(Total Files): cumulative 0, interval 0 2023-10-12T22:30:35.293 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: AddFile(L0 Files): cumulative 0, interval 0 2023-10-12T22:30:35.293 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: AddFile(Keys): cumulative 0, interval 0 2023-10-12T22:30:35.293 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Cumulative compaction: 0.00 GB write, 29.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-12T22:30:35.293 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Interval compaction: 0.00 GB write, 29.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-12T22:30:35.293 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-12T22:30:35.294 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.294 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ** File Read Latency Histogram By Level [default] ** 2023-10-12T22:30:35.294 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.294 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ** Compaction Stats [default] ** 2023-10-12T22:30:35.294 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-10-12T22:30:35.295 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-12T22:30:35.295 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: L0 2/0 87.89 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.5 0.00 0.00 1 0.001 0 0 2023-10-12T22:30:35.295 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Sum 2/0 87.89 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.5 0.00 0.00 1 0.001 0 0 2023-10-12T22:30:35.295 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-10-12T22:30:35.295 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.295 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ** Compaction Stats [default] ** 2023-10-12T22:30:35.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-10-12T22:30:35.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-12T22:30:35.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.5 0.00 0.00 1 0.001 0 0 2023-10-12T22:30:35.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Uptime(secs): 0.0 total, 0.0 interval 2023-10-12T22:30:35.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Flush(GB): cumulative 0.000, interval 0.000 2023-10-12T22:30:35.297 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-12T22:30:35.297 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: AddFile(Total Files): cumulative 0, interval 0 2023-10-12T22:30:35.297 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: AddFile(L0 Files): cumulative 0, interval 0 2023-10-12T22:30:35.297 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: AddFile(Keys): cumulative 0, interval 0 2023-10-12T22:30:35.297 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Cumulative compaction: 0.00 GB write, 29.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-12T22:30:35.298 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-12T22:30:35.298 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-12T22:30:35.298 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.298 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ** File Read Latency Histogram By Level [default] ** 2023-10-12T22:30:35.298 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.299 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.914+0000 7f7ee0eea700 0 starting mon.smithi131 rank 0 at public addrs [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] at bind addrs [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi131 fsid e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:30:35.299 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.915+0000 7f7ee0eea700 1 mon.smithi131@-1(???) e1 preinit fsid e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:30:35.299 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.915+0000 7f7ee0eea700 0 mon.smithi131@-1(???).mds e1 new map 2023-10-12T22:30:35.299 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.915+0000 7f7ee0eea700 0 mon.smithi131@-1(???).mds e1 print_map 2023-10-12T22:30:35.299 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: e1 2023-10-12T22:30:35.299 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: enable_multiple, ever_enabled_multiple: 1,1 2023-10-12T22:30:35.300 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-10-12T22:30:35.300 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: legacy client fscid: -1 2023-10-12T22:30:35.300 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.300 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: No filesystems configured 2023-10-12T22:30:35.300 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: 2023-10-12T22:30:35.301 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.915+0000 7f7ee0eea700 0 mon.smithi131@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-10-12T22:30:35.301 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.915+0000 7f7ee0eea700 0 mon.smithi131@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-12T22:30:35.301 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.915+0000 7f7ee0eea700 0 mon.smithi131@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-12T22:30:35.301 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.915+0000 7f7ee0eea700 0 mon.smithi131@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-12T22:30:35.301 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:32.995111+0000 mon.smithi131 (mon.0) 8 : audit [INF] from='client.? 172.21.15.131:0/1232548477' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-10-12T22:30:35.302 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:32.998281+0000 mon.smithi131 (mon.0) 9 : audit [INF] from='client.? 172.21.15.131:0/1232548477' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-10-12T22:30:35.302 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 1 mon.smithi131@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-10-12T22:30:35.302 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 20 expand_channel_meta expand map: {default=false} 2023-10-12T22:30:35.302 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 20 expand_channel_meta from 'false' to 'false' 2023-10-12T22:30:35.302 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 20 expand_channel_meta expanded map: {default=false} 2023-10-12T22:30:35.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 20 expand_channel_meta expand map: {default=info} 2023-10-12T22:30:35.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 20 expand_channel_meta from 'info' to 'info' 2023-10-12T22:30:35.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 20 expand_channel_meta expanded map: {default=info} 2023-10-12T22:30:35.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 20 expand_channel_meta expand map: {default=daemon} 2023-10-12T22:30:35.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-10-12T22:30:35.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 20 expand_channel_meta expanded map: {default=daemon} 2023-10-12T22:30:35.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 20 expand_channel_meta expand map: {default=debug} 2023-10-12T22:30:35.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 20 expand_channel_meta from 'debug' to 'debug' 2023-10-12T22:30:35.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:30:34.916+0000 7f7ee0eea700 20 expand_channel_meta expanded map: {default=debug} 2023-10-12T22:30:35.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:34.933068+0000 mon.smithi131 (mon.0) 1 : cluster [INF] mon.smithi131 is new leader, mons smithi131 in quorum (ranks 0) 2023-10-12T22:30:35.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:34.933179+0000 mon.smithi131 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi131=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} 2023-10-12T22:30:35.305 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:34.933250+0000 mon.smithi131 (mon.0) 3 : cluster [DBG] fsmap 2023-10-12T22:30:35.305 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:34.933292+0000 mon.smithi131 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-10-12T22:30:35.305 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:34.933800+0000 mon.smithi131 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-10-12T22:30:35.309 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.277+0000 7f2c50fb5700 1 Processor -- start 2023-10-12T22:30:35.309 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.277+0000 7f2c50fb5700 1 -- start start 2023-10-12T22:30:35.309 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.277+0000 7f2c50fb5700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 0x7f2c4c070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:35.309 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.277+0000 7f2c50fb5700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c0720f0 con 0x7f2c4c071cd0 2023-10-12T22:30:35.309 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.278+0000 7f2c4b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 0x7f2c4c070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:35.310 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.278+0000 7f2c4b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 0x7f2c4c070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48440/0 (socket says 172.21.15.131:48440) 2023-10-12T22:30:35.310 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.278+0000 7f2c4b7fe700 1 -- 172.21.15.131:0/4292467522 learned_addr learned my addr 172.21.15.131:0/4292467522 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:35.310 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.278+0000 7f2c4b7fe700 1 -- 172.21.15.131:0/4292467522 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c070760 con 0x7f2c4c071cd0 2023-10-12T22:30:35.310 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.278+0000 7f2c4b7fe700 1 --2- 172.21.15.131:0/4292467522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 0x7f2c4c070270 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f2c3c00b5f0 tx=0x7f2c3c006ee0).ready entity=mon.0 client_cookie=1aecf5dc0a4f02dd server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:35.310 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.278+0000 7f2c4a7fc700 1 -- 172.21.15.131:0/4292467522 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c3c00d600 con 0x7f2c4c071cd0 2023-10-12T22:30:35.311 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.278+0000 7f2c4a7fc700 1 -- 172.21.15.131:0/4292467522 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2c3c00dbd0 con 0x7f2c4c071cd0 2023-10-12T22:30:35.311 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.278+0000 7f2c4a7fc700 1 -- 172.21.15.131:0/4292467522 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c3c011c00 con 0x7f2c4c071cd0 2023-10-12T22:30:35.311 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.279+0000 7f2c50fb5700 1 -- 172.21.15.131:0/4292467522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 msgr2=0x7f2c4c070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:35.311 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.279+0000 7f2c50fb5700 1 --2- 172.21.15.131:0/4292467522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 0x7f2c4c070270 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f2c3c00b5f0 tx=0x7f2c3c006ee0).stop 2023-10-12T22:30:35.311 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.279+0000 7f2c50fb5700 1 -- 172.21.15.131:0/4292467522 shutdown_connections 2023-10-12T22:30:35.312 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.279+0000 7f2c50fb5700 1 --2- 172.21.15.131:0/4292467522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 0x7f2c4c070270 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:35.312 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.279+0000 7f2c50fb5700 1 -- 172.21.15.131:0/4292467522 >> 172.21.15.131:0/4292467522 conn(0x7f2c4c0f7200 msgr2=0x7f2c4c0f9620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:35.312 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.279+0000 7f2c50fb5700 1 -- 172.21.15.131:0/4292467522 shutdown_connections 2023-10-12T22:30:35.312 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.279+0000 7f2c50fb5700 1 -- 172.21.15.131:0/4292467522 wait complete. 2023-10-12T22:30:35.312 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.280+0000 7f2c50fb5700 1 Processor -- start 2023-10-12T22:30:35.313 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.280+0000 7f2c50fb5700 1 -- start start 2023-10-12T22:30:35.313 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.280+0000 7f2c50fb5700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 0x7f2c4c10c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:35.313 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.280+0000 7f2c50fb5700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c10ce60 con 0x7f2c4c071cd0 2023-10-12T22:30:35.313 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.280+0000 7f2c4b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 0x7f2c4c10c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:35.313 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.280+0000 7f2c4b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 0x7f2c4c10c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48442/0 (socket says 172.21.15.131:48442) 2023-10-12T22:30:35.314 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.280+0000 7f2c4b7fe700 1 -- 172.21.15.131:0/3518536094 learned_addr learned my addr 172.21.15.131:0/3518536094 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:35.314 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.281+0000 7f2c4b7fe700 1 -- 172.21.15.131:0/3518536094 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c00b200 con 0x7f2c4c071cd0 2023-10-12T22:30:35.314 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.281+0000 7f2c4b7fe700 1 --2- 172.21.15.131:0/3518536094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 0x7f2c4c10c970 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2c3c00b190 tx=0x7f2c3c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:35.314 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.281+0000 7f2c33fff700 1 -- 172.21.15.131:0/3518536094 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c3c01b410 con 0x7f2c4c071cd0 2023-10-12T22:30:35.314 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.281+0000 7f2c33fff700 1 -- 172.21.15.131:0/3518536094 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2c3c01b9e0 con 0x7f2c4c071cd0 2023-10-12T22:30:35.315 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.281+0000 7f2c50fb5700 1 -- 172.21.15.131:0/3518536094 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4c10d060 con 0x7f2c4c071cd0 2023-10-12T22:30:35.315 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.281+0000 7f2c33fff700 1 -- 172.21.15.131:0/3518536094 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c3c01a880 con 0x7f2c4c071cd0 2023-10-12T22:30:35.315 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.281+0000 7f2c50fb5700 1 -- 172.21.15.131:0/3518536094 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4c10d470 con 0x7f2c4c071cd0 2023-10-12T22:30:35.315 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.281+0000 7f2c33fff700 1 -- 172.21.15.131:0/3518536094 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f2c3c00f070 con 0x7f2c4c071cd0 2023-10-12T22:30:35.315 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.281+0000 7f2c33fff700 1 -- 172.21.15.131:0/3518536094 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2c3c025b70 con 0x7f2c4c071cd0 2023-10-12T22:30:35.316 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.282+0000 7f2c50fb5700 1 -- 172.21.15.131:0/3518536094 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4c045ad0 con 0x7f2c4c071cd0 2023-10-12T22:30:35.316 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.284+0000 7f2c33fff700 1 -- 172.21.15.131:0/3518536094 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2c3c016070 con 0x7f2c4c071cd0 2023-10-12T22:30:35.349 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.348+0000 7f2c50fb5700 1 -- 172.21.15.131:0/3518536094 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f2c4c106810 con 0x7f2c4c071cd0 2023-10-12T22:30:35.354 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.353+0000 7f2c33fff700 1 -- 172.21.15.131:0/3518536094 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2c3c025db0 con 0x7f2c4c071cd0 2023-10-12T22:30:35.355 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.353+0000 7f2c50fb5700 1 -- 172.21.15.131:0/3518536094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 msgr2=0x7f2c4c10c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:35.355 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.353+0000 7f2c50fb5700 1 --2- 172.21.15.131:0/3518536094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 0x7f2c4c10c970 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2c3c00b190 tx=0x7f2c3c006ee0).stop 2023-10-12T22:30:35.355 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.353+0000 7f2c50fb5700 1 -- 172.21.15.131:0/3518536094 shutdown_connections 2023-10-12T22:30:35.356 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.353+0000 7f2c50fb5700 1 --2- 172.21.15.131:0/3518536094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c4c071cd0 0x7f2c4c10c970 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:35.356 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.353+0000 7f2c50fb5700 1 -- 172.21.15.131:0/3518536094 >> 172.21.15.131:0/3518536094 conn(0x7f2c4c0f7200 msgr2=0x7f2c4c0f8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:35.356 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.354+0000 7f2c50fb5700 1 -- 172.21.15.131:0/3518536094 shutdown_connections 2023-10-12T22:30:35.356 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:35.354+0000 7f2c50fb5700 1 -- 172.21.15.131:0/3518536094 wait complete. 2023-10-12T22:30:35.702 INFO:teuthology.orchestra.run.smithi131.stderr:Wrote config to /etc/ceph/ceph.conf 2023-10-12T22:30:35.702 INFO:teuthology.orchestra.run.smithi131.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-10-12T22:30:35.702 INFO:teuthology.orchestra.run.smithi131.stderr:Creating mgr... 2023-10-12T22:30:35.703 INFO:teuthology.orchestra.run.smithi131.stderr:Verifying port 9283 ... 2023-10-12T22:30:35.704 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl daemon-reload 2023-10-12T22:30:35.802 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl stop ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mgr.smithi131.efktma 2023-10-12T22:30:35.824 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl reset-failed ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mgr.smithi131.efktma 2023-10-12T22:30:35.830 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Failed to reset failed state of unit ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mgr.smithi131.efktma.service: Unit ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mgr.smithi131.efktma.service not loaded. 2023-10-12T22:30:35.831 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl enable ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mgr.smithi131.efktma 2023-10-12T22:30:35.836 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Created symlink /etc/systemd/system/ceph-e4069eba-694e-11ee-8db6-212e2dc638e7.target.wants/ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mgr.smithi131.efktma.service → /etc/systemd/system/ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@.service. 2023-10-12T22:30:35.922 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl start ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mgr.smithi131.efktma 2023-10-12T22:30:36.236 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-enabled firewalld.service 2023-10-12T22:30:36.246 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: disabled 2023-10-12T22:30:36.247 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-active firewalld.service 2023-10-12T22:30:36.253 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: inactive 2023-10-12T22:30:36.254 INFO:teuthology.orchestra.run.smithi131.stderr:firewalld.service is not enabled 2023-10-12T22:30:36.255 INFO:teuthology.orchestra.run.smithi131.stderr:Not possible to enable service . firewalld.service is not available 2023-10-12T22:30:36.255 INFO:teuthology.orchestra.run.smithi131.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-10-12T22:30:36.255 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-enabled firewalld.service 2023-10-12T22:30:36.263 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: disabled 2023-10-12T22:30:36.264 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-active firewalld.service 2023-10-12T22:30:36.271 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: inactive 2023-10-12T22:30:36.272 INFO:teuthology.orchestra.run.smithi131.stderr:firewalld.service is not enabled 2023-10-12T22:30:36.272 INFO:teuthology.orchestra.run.smithi131.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-10-12T22:30:36.272 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for mgr to start... 2023-10-12T22:30:36.273 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for mgr... 2023-10-12T22:30:36.273 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-10-12T22:30:36.622 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.617+0000 7f58a5a99700 1 Processor -- start 2023-10-12T22:30:36.622 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.617+0000 7f58a5a99700 1 -- start start 2023-10-12T22:30:36.623 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.617+0000 7f58a5a99700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 0x7f58a00faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:36.623 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.617+0000 7f58a5a99700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a00fb480 con 0x7f58a00faae0 2023-10-12T22:30:36.623 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.618+0000 7f589f7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 0x7f58a00faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:36.623 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.618+0000 7f589f7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 0x7f58a00faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48466/0 (socket says 172.21.15.131:48466) 2023-10-12T22:30:36.623 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.618+0000 7f589f7fe700 1 -- 172.21.15.131:0/2804641087 learned_addr learned my addr 172.21.15.131:0/2804641087 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:36.624 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.618+0000 7f589f7fe700 1 -- 172.21.15.131:0/2804641087 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a00fb5c0 con 0x7f58a00faae0 2023-10-12T22:30:36.624 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.618+0000 7f589f7fe700 1 --2- 172.21.15.131:0/2804641087 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 0x7f58a00faf00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f589000b5f0 tx=0x7f5890006ee0).ready entity=mon.0 client_cookie=8882f48878fee087 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:36.624 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.619+0000 7f589e7fc700 1 -- 172.21.15.131:0/2804641087 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f589000d600 con 0x7f58a00faae0 2023-10-12T22:30:36.624 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.619+0000 7f589e7fc700 1 -- 172.21.15.131:0/2804641087 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f589000dbd0 con 0x7f58a00faae0 2023-10-12T22:30:36.624 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.619+0000 7f589e7fc700 1 -- 172.21.15.131:0/2804641087 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5890011c00 con 0x7f58a00faae0 2023-10-12T22:30:36.625 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.619+0000 7f58a5a99700 1 -- 172.21.15.131:0/2804641087 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 msgr2=0x7f58a00faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:36.625 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.619+0000 7f58a5a99700 1 --2- 172.21.15.131:0/2804641087 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 0x7f58a00faf00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f589000b5f0 tx=0x7f5890006ee0).stop 2023-10-12T22:30:36.625 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.620+0000 7f58a5a99700 1 -- 172.21.15.131:0/2804641087 shutdown_connections 2023-10-12T22:30:36.625 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.620+0000 7f58a5a99700 1 --2- 172.21.15.131:0/2804641087 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 0x7f58a00faf00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:36.625 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.620+0000 7f58a5a99700 1 -- 172.21.15.131:0/2804641087 >> 172.21.15.131:0/2804641087 conn(0x7f58a00f6280 msgr2=0x7f58a00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:36.625 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.620+0000 7f58a5a99700 1 -- 172.21.15.131:0/2804641087 shutdown_connections 2023-10-12T22:30:36.626 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.620+0000 7f58a5a99700 1 -- 172.21.15.131:0/2804641087 wait complete. 2023-10-12T22:30:36.626 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.620+0000 7f58a5a99700 1 Processor -- start 2023-10-12T22:30:36.626 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.620+0000 7f58a5a99700 1 -- start start 2023-10-12T22:30:36.626 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.621+0000 7f58a5a99700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 0x7f58a010aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:36.626 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.621+0000 7f58a5a99700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a010af00 con 0x7f58a00faae0 2023-10-12T22:30:36.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.621+0000 7f589f7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 0x7f58a010aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:36.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.621+0000 7f589f7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 0x7f58a010aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48472/0 (socket says 172.21.15.131:48472) 2023-10-12T22:30:36.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.621+0000 7f589f7fe700 1 -- 172.21.15.131:0/2446457699 learned_addr learned my addr 172.21.15.131:0/2446457699 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:36.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.621+0000 7f589f7fe700 1 -- 172.21.15.131:0/2446457699 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589000b200 con 0x7f58a00faae0 2023-10-12T22:30:36.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.621+0000 7f589f7fe700 1 --2- 172.21.15.131:0/2446457699 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 0x7f58a010aa10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f589000b1d0 tx=0x7f5890006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:36.628 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.621+0000 7f5887fff700 1 -- 172.21.15.131:0/2446457699 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5890011a20 con 0x7f58a00faae0 2023-10-12T22:30:36.628 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.621+0000 7f5887fff700 1 -- 172.21.15.131:0/2446457699 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f589001b880 con 0x7f58a00faae0 2023-10-12T22:30:36.629 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.621+0000 7f5887fff700 1 -- 172.21.15.131:0/2446457699 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f589001a700 con 0x7f58a00faae0 2023-10-12T22:30:36.629 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.621+0000 7f58a5a99700 1 -- 172.21.15.131:0/2446457699 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a010b100 con 0x7f58a00faae0 2023-10-12T22:30:36.629 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.621+0000 7f58a5a99700 1 -- 172.21.15.131:0/2446457699 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a010b510 con 0x7f58a00faae0 2023-10-12T22:30:36.629 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.622+0000 7f5887fff700 1 -- 172.21.15.131:0/2446457699 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f589000f070 con 0x7f58a00faae0 2023-10-12T22:30:36.629 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.622+0000 7f5887fff700 1 -- 172.21.15.131:0/2446457699 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f58900259b0 con 0x7f58a00faae0 2023-10-12T22:30:36.629 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.623+0000 7f58a5a99700 1 -- 172.21.15.131:0/2446457699 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a0045ad0 con 0x7f58a00faae0 2023-10-12T22:30:36.630 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.625+0000 7f5887fff700 1 -- 172.21.15.131:0/2446457699 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5890016070 con 0x7f58a00faae0 2023-10-12T22:30:36.712 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.709+0000 7f58a5a99700 1 -- 172.21.15.131:0/2446457699 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f58a00fc8c0 con 0x7f58a00faae0 2023-10-12T22:30:36.712 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.710+0000 7f5887fff700 1 -- 172.21.15.131:0/2446457699 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f589001bab0 con 0x7f58a00faae0 2023-10-12T22:30:36.712 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.712 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2023-10-12T22:30:36.713 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "fsid": "e4069eba-694e-11ee-8db6-212e2dc638e7", 2023-10-12T22:30:36.713 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "health": { 2023-10-12T22:30:36.713 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-10-12T22:30:36.713 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "checks": {}, 2023-10-12T22:30:36.713 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mutes": [] 2023-10-12T22:30:36.714 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:36.714 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-10-12T22:30:36.714 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum": [ 2023-10-12T22:30:36.719 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 0 2023-10-12T22:30:36.719 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2023-10-12T22:30:36.719 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum_names": [ 2023-10-12T22:30:36.720 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "smithi131" 2023-10-12T22:30:36.720 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2023-10-12T22:30:36.720 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-10-12T22:30:36.720 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "monmap": { 2023-10-12T22:30:36.721 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2023-10-12T22:30:36.721 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-10-12T22:30:36.721 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_mons": 1 2023-10-12T22:30:36.721 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:36.721 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osdmap": { 2023-10-12T22:30:36.722 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2023-10-12T22:30:36.722 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_osds": 0, 2023-10-12T22:30:36.722 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-10-12T22:30:36.722 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-10-12T22:30:36.722 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-10-12T22:30:36.722 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-10-12T22:30:36.723 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-10-12T22:30:36.723 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:36.723 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "pgmap": { 2023-10-12T22:30:36.723 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-10-12T22:30:36.723 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-10-12T22:30:36.724 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_pools": 0, 2023-10-12T22:30:36.724 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_objects": 0, 2023-10-12T22:30:36.724 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-10-12T22:30:36.724 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-10-12T22:30:36.724 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-10-12T22:30:36.724 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_total": 0 2023-10-12T22:30:36.725 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:36.725 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "fsmap": { 2023-10-12T22:30:36.725 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2023-10-12T22:30:36.725 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "by_rank": [], 2023-10-12T22:30:36.725 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "up:standby": 0 2023-10-12T22:30:36.726 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:36.726 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mgrmap": { 2023-10-12T22:30:36.726 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "available": false, 2023-10-12T22:30:36.726 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-10-12T22:30:36.726 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "modules": [ 2023-10-12T22:30:36.726 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "iostat", 2023-10-12T22:30:36.727 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "restful" 2023-10-12T22:30:36.727 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2023-10-12T22:30:36.727 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "services": {} 2023-10-12T22:30:36.727 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:36.727 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "servicemap": { 2023-10-12T22:30:36.728 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2023-10-12T22:30:36.728 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "modified": "2023-10-12T22:30:31.736889+0000", 2023-10-12T22:30:36.728 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "services": {} 2023-10-12T22:30:36.728 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:36.728 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "progress_events": {} 2023-10-12T22:30:36.728 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2023-10-12T22:30:36.729 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.711+0000 7f58a5a99700 1 -- 172.21.15.131:0/2446457699 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 msgr2=0x7f58a010aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:36.729 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.711+0000 7f58a5a99700 1 --2- 172.21.15.131:0/2446457699 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 0x7f58a010aa10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f589000b1d0 tx=0x7f5890006ee0).stop 2023-10-12T22:30:36.729 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.711+0000 7f58a5a99700 1 -- 172.21.15.131:0/2446457699 shutdown_connections 2023-10-12T22:30:36.729 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.711+0000 7f58a5a99700 1 --2- 172.21.15.131:0/2446457699 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a00faae0 0x7f58a010aa10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:36.729 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.711+0000 7f58a5a99700 1 -- 172.21.15.131:0/2446457699 >> 172.21.15.131:0/2446457699 conn(0x7f58a00f6280 msgr2=0x7f58a00f6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:36.730 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.711+0000 7f58a5a99700 1 -- 172.21.15.131:0/2446457699 shutdown_connections 2023-10-12T22:30:36.730 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:36.711+0000 7f58a5a99700 1 -- 172.21.15.131:0/2446457699 wait complete. 2023-10-12T22:30:36.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:36 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:35.354164+0000 mon.smithi131 (mon.0) 6 : audit [INF] from='client.? 172.21.15.131:0/3518536094' entity='client.admin' 2023-10-12T22:30:37.094 INFO:teuthology.orchestra.run.smithi131.stderr:mgr not available, waiting (1/15)... 2023-10-12T22:30:37.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:37 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:36.711212+0000 mon.smithi131 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.131:0/2446457699' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-12T22:30:39.096 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-10-12T22:30:39.475 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.473+0000 7f458419b700 1 Processor -- start 2023-10-12T22:30:39.479 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.474+0000 7f458419b700 1 -- start start 2023-10-12T22:30:39.479 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.474+0000 7f458419b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 0x7f457c0fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:39.479 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.474+0000 7f458419b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c0fed20 con 0x7f457c06fcd0 2023-10-12T22:30:39.479 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.474+0000 7f4581f37700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 0x7f457c0fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:39.480 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.474+0000 7f4581f37700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 0x7f457c0fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:49072/0 (socket says 172.21.15.131:49072) 2023-10-12T22:30:39.480 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.474+0000 7f4581f37700 1 -- 172.21.15.131:0/1841106880 learned_addr learned my addr 172.21.15.131:0/1841106880 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:39.480 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.474+0000 7f4581f37700 1 -- 172.21.15.131:0/1841106880 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c0fee60 con 0x7f457c06fcd0 2023-10-12T22:30:39.480 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.474+0000 7f4581f37700 1 --2- 172.21.15.131:0/1841106880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 0x7f457c0fe7a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f457800b5f0 tx=0x7f4578006ee0).ready entity=mon.0 client_cookie=8ee64eaba8d6de32 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:39.480 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.475+0000 7f4580f35700 1 -- 172.21.15.131:0/1841106880 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f457800d600 con 0x7f457c06fcd0 2023-10-12T22:30:39.481 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.475+0000 7f4580f35700 1 -- 172.21.15.131:0/1841106880 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f457800dbd0 con 0x7f457c06fcd0 2023-10-12T22:30:39.481 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.475+0000 7f4580f35700 1 -- 172.21.15.131:0/1841106880 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4578011c00 con 0x7f457c06fcd0 2023-10-12T22:30:39.481 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.475+0000 7f458419b700 1 -- 172.21.15.131:0/1841106880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 msgr2=0x7f457c0fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:39.481 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.475+0000 7f458419b700 1 --2- 172.21.15.131:0/1841106880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 0x7f457c0fe7a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f457800b5f0 tx=0x7f4578006ee0).stop 2023-10-12T22:30:39.481 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.475+0000 7f458419b700 1 -- 172.21.15.131:0/1841106880 shutdown_connections 2023-10-12T22:30:39.482 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.475+0000 7f458419b700 1 --2- 172.21.15.131:0/1841106880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 0x7f457c0fe7a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:39.482 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.475+0000 7f458419b700 1 -- 172.21.15.131:0/1841106880 >> 172.21.15.131:0/1841106880 conn(0x7f457c0f6280 msgr2=0x7f457c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:39.482 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.476+0000 7f458419b700 1 -- 172.21.15.131:0/1841106880 shutdown_connections 2023-10-12T22:30:39.482 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.476+0000 7f458419b700 1 -- 172.21.15.131:0/1841106880 wait complete. 2023-10-12T22:30:39.483 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.476+0000 7f458419b700 1 Processor -- start 2023-10-12T22:30:39.483 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.476+0000 7f458419b700 1 -- start start 2023-10-12T22:30:39.483 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.476+0000 7f458419b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 0x7f457c10a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:39.483 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.476+0000 7f458419b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c10ae90 con 0x7f457c06fcd0 2023-10-12T22:30:39.483 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.476+0000 7f4581f37700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 0x7f457c10a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:39.484 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.477+0000 7f4581f37700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 0x7f457c10a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:49086/0 (socket says 172.21.15.131:49086) 2023-10-12T22:30:39.484 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.477+0000 7f4581f37700 1 -- 172.21.15.131:0/2236678801 learned_addr learned my addr 172.21.15.131:0/2236678801 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:39.484 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.477+0000 7f4581f37700 1 -- 172.21.15.131:0/2236678801 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457800b200 con 0x7f457c06fcd0 2023-10-12T22:30:39.484 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.477+0000 7f4581f37700 1 --2- 172.21.15.131:0/2236678801 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 0x7f457c10a9a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f457800e040 tx=0x7f4578006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:39.484 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.477+0000 7f456a7fc700 1 -- 172.21.15.131:0/2236678801 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f457801b410 con 0x7f457c06fcd0 2023-10-12T22:30:39.485 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.477+0000 7f456a7fc700 1 -- 172.21.15.131:0/2236678801 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f457801b9e0 con 0x7f457c06fcd0 2023-10-12T22:30:39.485 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.477+0000 7f458419b700 1 -- 172.21.15.131:0/2236678801 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457c10b090 con 0x7f457c06fcd0 2023-10-12T22:30:39.485 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.477+0000 7f456a7fc700 1 -- 172.21.15.131:0/2236678801 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f457801a770 con 0x7f457c06fcd0 2023-10-12T22:30:39.485 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.477+0000 7f458419b700 1 -- 172.21.15.131:0/2236678801 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c10b4a0 con 0x7f457c06fcd0 2023-10-12T22:30:39.485 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.478+0000 7f456a7fc700 1 -- 172.21.15.131:0/2236678801 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f457800f070 con 0x7f457c06fcd0 2023-10-12T22:30:39.486 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.478+0000 7f456a7fc700 1 -- 172.21.15.131:0/2236678801 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4578024a80 con 0x7f457c06fcd0 2023-10-12T22:30:39.486 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.478+0000 7f458419b700 1 -- 172.21.15.131:0/2236678801 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457c045ad0 con 0x7f457c06fcd0 2023-10-12T22:30:39.486 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.480+0000 7f456a7fc700 1 -- 172.21.15.131:0/2236678801 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4578016070 con 0x7f457c06fcd0 2023-10-12T22:30:39.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.548+0000 7f458419b700 1 -- 172.21.15.131:0/2236678801 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f457c104820 con 0x7f457c06fcd0 2023-10-12T22:30:39.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.548+0000 7f456a7fc700 1 -- 172.21.15.131:0/2236678801 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f4578006a70 con 0x7f457c06fcd0 2023-10-12T22:30:39.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2023-10-12T22:30:39.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "fsid": "e4069eba-694e-11ee-8db6-212e2dc638e7", 2023-10-12T22:30:39.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "health": { 2023-10-12T22:30:39.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-10-12T22:30:39.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "checks": {}, 2023-10-12T22:30:39.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mutes": [] 2023-10-12T22:30:39.553 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:39.553 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-10-12T22:30:39.558 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum": [ 2023-10-12T22:30:39.558 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 0 2023-10-12T22:30:39.558 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2023-10-12T22:30:39.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum_names": [ 2023-10-12T22:30:39.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "smithi131" 2023-10-12T22:30:39.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2023-10-12T22:30:39.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-10-12T22:30:39.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "monmap": { 2023-10-12T22:30:39.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2023-10-12T22:30:39.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-10-12T22:30:39.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_mons": 1 2023-10-12T22:30:39.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:39.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osdmap": { 2023-10-12T22:30:39.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2023-10-12T22:30:39.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_osds": 0, 2023-10-12T22:30:39.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-10-12T22:30:39.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-10-12T22:30:39.562 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-10-12T22:30:39.562 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-10-12T22:30:39.562 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-10-12T22:30:39.562 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:39.562 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "pgmap": { 2023-10-12T22:30:39.563 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-10-12T22:30:39.563 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-10-12T22:30:39.563 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_pools": 0, 2023-10-12T22:30:39.563 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_objects": 0, 2023-10-12T22:30:39.563 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-10-12T22:30:39.564 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-10-12T22:30:39.564 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-10-12T22:30:39.564 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_total": 0 2023-10-12T22:30:39.564 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:39.564 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "fsmap": { 2023-10-12T22:30:39.565 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2023-10-12T22:30:39.565 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "by_rank": [], 2023-10-12T22:30:39.565 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "up:standby": 0 2023-10-12T22:30:39.565 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:39.565 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mgrmap": { 2023-10-12T22:30:39.565 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "available": false, 2023-10-12T22:30:39.566 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-10-12T22:30:39.566 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "modules": [ 2023-10-12T22:30:39.566 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "iostat", 2023-10-12T22:30:39.566 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "restful" 2023-10-12T22:30:39.566 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2023-10-12T22:30:39.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "services": {} 2023-10-12T22:30:39.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:39.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "servicemap": { 2023-10-12T22:30:39.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2023-10-12T22:30:39.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "modified": "2023-10-12T22:30:31.736889+0000", 2023-10-12T22:30:39.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "services": {} 2023-10-12T22:30:39.568 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:39.568 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "progress_events": {} 2023-10-12T22:30:39.568 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2023-10-12T22:30:39.568 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.550+0000 7f458419b700 1 -- 172.21.15.131:0/2236678801 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 msgr2=0x7f457c10a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:39.568 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.550+0000 7f458419b700 1 --2- 172.21.15.131:0/2236678801 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 0x7f457c10a9a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f457800e040 tx=0x7f4578006ee0).stop 2023-10-12T22:30:39.568 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.550+0000 7f458419b700 1 -- 172.21.15.131:0/2236678801 shutdown_connections 2023-10-12T22:30:39.569 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.550+0000 7f458419b700 1 --2- 172.21.15.131:0/2236678801 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f457c06fcd0 0x7f457c10a9a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:39.569 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.550+0000 7f458419b700 1 -- 172.21.15.131:0/2236678801 >> 172.21.15.131:0/2236678801 conn(0x7f457c0f6280 msgr2=0x7f457c0f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:39.569 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.550+0000 7f458419b700 1 -- 172.21.15.131:0/2236678801 shutdown_connections 2023-10-12T22:30:39.569 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:39.550+0000 7f458419b700 1 -- 172.21.15.131:0/2236678801 wait complete. 2023-10-12T22:30:39.892 INFO:teuthology.orchestra.run.smithi131.stderr:mgr not available, waiting (2/15)... 2023-10-12T22:30:39.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:39 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:39.549799+0000 mon.smithi131 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.131:0/2236678801' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-12T22:30:41.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:41.204058+0000 mon.smithi131 (mon.0) 9 : cluster [INF] Activating manager daemon smithi131.efktma 2023-10-12T22:30:41.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:41.206766+0000 mon.smithi131 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi131.efktma(active, starting, since 0.00281614s) 2023-10-12T22:30:41.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:41.207605+0000 mon.smithi131 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.131:0/3158966350' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-12T22:30:41.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:41.207777+0000 mon.smithi131 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.131:0/3158966350' entity='mgr.smithi131.efktma' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-12T22:30:41.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:41.207906+0000 mon.smithi131 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.131:0/3158966350' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-12T22:30:41.741 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:41.208027+0000 mon.smithi131 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.131:0/3158966350' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2023-10-12T22:30:41.741 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:41.208265+0000 mon.smithi131 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.131:0/3158966350' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mgr metadata", "who": "smithi131.efktma", "id": "smithi131.efktma"}]: dispatch 2023-10-12T22:30:41.741 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:41.221615+0000 mon.smithi131 (mon.0) 16 : cluster [INF] Manager daemon smithi131.efktma is now available 2023-10-12T22:30:41.741 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:41.229166+0000 mon.smithi131 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.131:0/3158966350' entity='mgr.smithi131.efktma' 2023-10-12T22:30:41.741 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:41.232834+0000 mon.smithi131 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.131:0/3158966350' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/mirror_snapshot_schedule"}]: dispatch 2023-10-12T22:30:41.742 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:41.233932+0000 mon.smithi131 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.131:0/3158966350' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/trash_purge_schedule"}]: dispatch 2023-10-12T22:30:41.742 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:41.237941+0000 mon.smithi131 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.131:0/3158966350' entity='mgr.smithi131.efktma' 2023-10-12T22:30:41.742 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:41.239952+0000 mon.smithi131 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.131:0/3158966350' entity='mgr.smithi131.efktma' 2023-10-12T22:30:41.894 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-10-12T22:30:42.223 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.221+0000 7fb77b4ff700 1 Processor -- start 2023-10-12T22:30:42.223 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.221+0000 7fb77b4ff700 1 -- start start 2023-10-12T22:30:42.223 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.221+0000 7fb77b4ff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 0x7fb774100c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:42.224 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.221+0000 7fb77b4ff700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740fc6a0 con 0x7fb77406fc50 2023-10-12T22:30:42.224 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.222+0000 7fb77929b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 0x7fb774100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:42.224 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.222+0000 7fb77929b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 0x7fb774100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:49150/0 (socket says 172.21.15.131:49150) 2023-10-12T22:30:42.224 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.222+0000 7fb77929b700 1 -- 172.21.15.131:0/4256984264 learned_addr learned my addr 172.21.15.131:0/4256984264 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:42.226 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.222+0000 7fb77929b700 1 -- 172.21.15.131:0/4256984264 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb774101110 con 0x7fb77406fc50 2023-10-12T22:30:42.226 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.222+0000 7fb77929b700 1 --2- 172.21.15.131:0/4256984264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 0x7fb774100c20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb76c00b5f0 tx=0x7fb76c006ee0).ready entity=mon.0 client_cookie=feb572d2fa390165 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:42.226 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.222+0000 7fb773fff700 1 -- 172.21.15.131:0/4256984264 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb76c00d600 con 0x7fb77406fc50 2023-10-12T22:30:42.226 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.222+0000 7fb773fff700 1 -- 172.21.15.131:0/4256984264 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb76c00dbd0 con 0x7fb77406fc50 2023-10-12T22:30:42.227 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.222+0000 7fb773fff700 1 -- 172.21.15.131:0/4256984264 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb76c011c00 con 0x7fb77406fc50 2023-10-12T22:30:42.227 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.223+0000 7fb77b4ff700 1 -- 172.21.15.131:0/4256984264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 msgr2=0x7fb774100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:42.227 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.223+0000 7fb77b4ff700 1 --2- 172.21.15.131:0/4256984264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 0x7fb774100c20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb76c00b5f0 tx=0x7fb76c006ee0).stop 2023-10-12T22:30:42.227 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.223+0000 7fb77b4ff700 1 -- 172.21.15.131:0/4256984264 shutdown_connections 2023-10-12T22:30:42.227 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.223+0000 7fb77b4ff700 1 --2- 172.21.15.131:0/4256984264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 0x7fb774100c20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:42.228 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.223+0000 7fb77b4ff700 1 -- 172.21.15.131:0/4256984264 >> 172.21.15.131:0/4256984264 conn(0x7fb7740f6280 msgr2=0x7fb7740f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:42.228 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.223+0000 7fb77b4ff700 1 -- 172.21.15.131:0/4256984264 shutdown_connections 2023-10-12T22:30:42.228 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.223+0000 7fb77b4ff700 1 -- 172.21.15.131:0/4256984264 wait complete. 2023-10-12T22:30:42.228 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.223+0000 7fb77b4ff700 1 Processor -- start 2023-10-12T22:30:42.228 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.223+0000 7fb77b4ff700 1 -- start start 2023-10-12T22:30:42.229 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.224+0000 7fb77b4ff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 0x7fb77410ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:42.229 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.224+0000 7fb77b4ff700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77410d1b0 con 0x7fb77406fc50 2023-10-12T22:30:42.230 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.224+0000 7fb77929b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 0x7fb77410ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:42.230 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.224+0000 7fb77929b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 0x7fb77410ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:49160/0 (socket says 172.21.15.131:49160) 2023-10-12T22:30:42.230 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.224+0000 7fb77929b700 1 -- 172.21.15.131:0/621097872 learned_addr learned my addr 172.21.15.131:0/621097872 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:42.230 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.224+0000 7fb77929b700 1 -- 172.21.15.131:0/621097872 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c00b200 con 0x7fb77406fc50 2023-10-12T22:30:42.230 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.224+0000 7fb77929b700 1 --2- 172.21.15.131:0/621097872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 0x7fb77410ccc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fb76c006c70 tx=0x7fb76c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:42.230 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.224+0000 7fb771ffb700 1 -- 172.21.15.131:0/621097872 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb76c011690 con 0x7fb77406fc50 2023-10-12T22:30:42.231 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.224+0000 7fb77b4ff700 1 -- 172.21.15.131:0/621097872 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb77410d3b0 con 0x7fb77406fc50 2023-10-12T22:30:42.231 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.224+0000 7fb771ffb700 1 -- 172.21.15.131:0/621097872 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb76c01b880 con 0x7fb77406fc50 2023-10-12T22:30:42.231 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.224+0000 7fb771ffb700 1 -- 172.21.15.131:0/621097872 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb76c01a650 con 0x7fb77406fc50 2023-10-12T22:30:42.231 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.224+0000 7fb77b4ff700 1 -- 172.21.15.131:0/621097872 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77410d7c0 con 0x7fb77406fc50 2023-10-12T22:30:42.231 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.225+0000 7fb771ffb700 1 -- 172.21.15.131:0/621097872 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fb76c00f070 con 0x7fb77406fc50 2023-10-12T22:30:42.232 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.225+0000 7fb771ffb700 1 --2- 172.21.15.131:0/621097872 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7fb76002fc50 0x7fb760032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:42.232 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.225+0000 7fb771ffb700 1 -- 172.21.15.131:0/621097872 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb76c041ee0 con 0x7fb77406fc50 2023-10-12T22:30:42.232 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.225+0000 7fb778a9a700 1 --2- 172.21.15.131:0/621097872 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7fb76002fc50 0x7fb760032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:42.232 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.225+0000 7fb77b4ff700 1 -- 172.21.15.131:0/621097872 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb774045ad0 con 0x7fb77406fc50 2023-10-12T22:30:42.232 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.226+0000 7fb778a9a700 1 --2- 172.21.15.131:0/621097872 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7fb76002fc50 0x7fb760032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fb7680096a0 tx=0x7fb768006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:42.233 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.230+0000 7fb771ffb700 1 -- 172.21.15.131:0/621097872 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb76c016070 con 0x7fb77406fc50 2023-10-12T22:30:42.436 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.433+0000 7fb77b4ff700 1 -- 172.21.15.131:0/621097872 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb774101de0 con 0x7fb77406fc50 2023-10-12T22:30:42.436 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.434+0000 7fb771ffb700 1 -- 172.21.15.131:0/621097872 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7fb774101de0 con 0x7fb77406fc50 2023-10-12T22:30:42.436 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.436 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2023-10-12T22:30:42.437 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "fsid": "e4069eba-694e-11ee-8db6-212e2dc638e7", 2023-10-12T22:30:42.437 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "health": { 2023-10-12T22:30:42.437 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-10-12T22:30:42.437 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "checks": {}, 2023-10-12T22:30:42.437 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mutes": [] 2023-10-12T22:30:42.438 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:42.438 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-10-12T22:30:42.438 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum": [ 2023-10-12T22:30:42.444 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 0 2023-10-12T22:30:42.444 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2023-10-12T22:30:42.444 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum_names": [ 2023-10-12T22:30:42.444 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "smithi131" 2023-10-12T22:30:42.444 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2023-10-12T22:30:42.445 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-10-12T22:30:42.445 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "monmap": { 2023-10-12T22:30:42.445 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2023-10-12T22:30:42.445 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-10-12T22:30:42.445 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_mons": 1 2023-10-12T22:30:42.446 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:42.446 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osdmap": { 2023-10-12T22:30:42.446 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2023-10-12T22:30:42.446 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_osds": 0, 2023-10-12T22:30:42.446 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-10-12T22:30:42.447 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-10-12T22:30:42.447 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-10-12T22:30:42.447 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-10-12T22:30:42.447 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-10-12T22:30:42.447 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:42.448 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "pgmap": { 2023-10-12T22:30:42.448 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-10-12T22:30:42.448 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-10-12T22:30:42.448 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_pools": 0, 2023-10-12T22:30:42.448 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_objects": 0, 2023-10-12T22:30:42.449 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-10-12T22:30:42.449 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-10-12T22:30:42.449 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-10-12T22:30:42.449 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_total": 0 2023-10-12T22:30:42.449 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:42.450 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "fsmap": { 2023-10-12T22:30:42.450 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2023-10-12T22:30:42.450 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "by_rank": [], 2023-10-12T22:30:42.450 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "up:standby": 0 2023-10-12T22:30:42.450 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:42.450 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mgrmap": { 2023-10-12T22:30:42.451 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "available": true, 2023-10-12T22:30:42.451 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-10-12T22:30:42.451 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "modules": [ 2023-10-12T22:30:42.451 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "iostat", 2023-10-12T22:30:42.451 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "restful" 2023-10-12T22:30:42.452 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2023-10-12T22:30:42.452 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "services": {} 2023-10-12T22:30:42.452 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:42.452 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "servicemap": { 2023-10-12T22:30:42.452 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2023-10-12T22:30:42.453 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "modified": "2023-10-12T22:30:31.736889+0000", 2023-10-12T22:30:42.453 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "services": {} 2023-10-12T22:30:42.453 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2023-10-12T22:30:42.453 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "progress_events": {} 2023-10-12T22:30:42.453 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2023-10-12T22:30:42.453 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.435+0000 7fb77b4ff700 1 -- 172.21.15.131:0/621097872 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7fb76002fc50 msgr2=0x7fb760032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:42.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.435+0000 7fb77b4ff700 1 --2- 172.21.15.131:0/621097872 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7fb76002fc50 0x7fb760032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fb7680096a0 tx=0x7fb768006b40).stop 2023-10-12T22:30:42.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.435+0000 7fb77b4ff700 1 -- 172.21.15.131:0/621097872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 msgr2=0x7fb77410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:42.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.435+0000 7fb77b4ff700 1 --2- 172.21.15.131:0/621097872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 0x7fb77410ccc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fb76c006c70 tx=0x7fb76c006ee0).stop 2023-10-12T22:30:42.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.435+0000 7fb77b4ff700 1 -- 172.21.15.131:0/621097872 shutdown_connections 2023-10-12T22:30:42.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.435+0000 7fb77b4ff700 1 --2- 172.21.15.131:0/621097872 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7fb76002fc50 0x7fb760032110 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:42.455 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.435+0000 7fb77b4ff700 1 --2- 172.21.15.131:0/621097872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb77406fc50 0x7fb77410ccc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:42.455 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.435+0000 7fb77b4ff700 1 -- 172.21.15.131:0/621097872 >> 172.21.15.131:0/621097872 conn(0x7fb7740f6280 msgr2=0x7fb7740f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:42.455 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.435+0000 7fb77b4ff700 1 -- 172.21.15.131:0/621097872 shutdown_connections 2023-10-12T22:30:42.455 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:42.435+0000 7fb77b4ff700 1 -- 172.21.15.131:0/621097872 wait complete. 2023-10-12T22:30:42.783 INFO:teuthology.orchestra.run.smithi131.stderr:mgr is available 2023-10-12T22:30:42.784 INFO:teuthology.orchestra.run.smithi131.stderr:Enabling cephadm module... 2023-10-12T22:30:42.784 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-10-12T22:30:43.251 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.113+0000 7fca20c95700 1 Processor -- start 2023-10-12T22:30:43.251 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.113+0000 7fca20c95700 1 -- start start 2023-10-12T22:30:43.251 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.113+0000 7fca20c95700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 0x7fca1c0fe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:43.251 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.113+0000 7fca20c95700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0fed80 con 0x7fca1c06fc70 2023-10-12T22:30:43.252 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.113+0000 7fca1a59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 0x7fca1c0fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:43.252 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.113+0000 7fca1a59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 0x7fca1c0fe800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:49164/0 (socket says 172.21.15.131:49164) 2023-10-12T22:30:43.252 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.113+0000 7fca1a59c700 1 -- 172.21.15.131:0/1726487863 learned_addr learned my addr 172.21.15.131:0/1726487863 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:43.252 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.114+0000 7fca1a59c700 1 -- 172.21.15.131:0/1726487863 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c0feec0 con 0x7fca1c06fc70 2023-10-12T22:30:43.252 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.114+0000 7fca1a59c700 1 --2- 172.21.15.131:0/1726487863 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 0x7fca1c0fe800 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fca0c00b5f0 tx=0x7fca0c006ee0).ready entity=mon.0 client_cookie=873d234fe0c039be server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:43.253 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.114+0000 7fca1959a700 1 -- 172.21.15.131:0/1726487863 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca0c00d600 con 0x7fca1c06fc70 2023-10-12T22:30:43.253 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.114+0000 7fca1959a700 1 -- 172.21.15.131:0/1726487863 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fca0c00dbd0 con 0x7fca1c06fc70 2023-10-12T22:30:43.253 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.114+0000 7fca1959a700 1 -- 172.21.15.131:0/1726487863 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca0c011c00 con 0x7fca1c06fc70 2023-10-12T22:30:43.253 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.114+0000 7fca20c95700 1 -- 172.21.15.131:0/1726487863 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 msgr2=0x7fca1c0fe800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:43.253 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.114+0000 7fca20c95700 1 --2- 172.21.15.131:0/1726487863 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 0x7fca1c0fe800 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fca0c00b5f0 tx=0x7fca0c006ee0).stop 2023-10-12T22:30:43.253 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.114+0000 7fca20c95700 1 -- 172.21.15.131:0/1726487863 shutdown_connections 2023-10-12T22:30:43.256 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.114+0000 7fca20c95700 1 --2- 172.21.15.131:0/1726487863 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 0x7fca1c0fe800 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:43.256 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.114+0000 7fca20c95700 1 -- 172.21.15.131:0/1726487863 >> 172.21.15.131:0/1726487863 conn(0x7fca1c0f6280 msgr2=0x7fca1c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:43.256 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.114+0000 7fca20c95700 1 -- 172.21.15.131:0/1726487863 shutdown_connections 2023-10-12T22:30:43.257 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.114+0000 7fca20c95700 1 -- 172.21.15.131:0/1726487863 wait complete. 2023-10-12T22:30:43.257 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.115+0000 7fca20c95700 1 Processor -- start 2023-10-12T22:30:43.257 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.115+0000 7fca20c95700 1 -- start start 2023-10-12T22:30:43.257 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.115+0000 7fca20c95700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 0x7fca1c10a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:43.257 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.115+0000 7fca20c95700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c10ae90 con 0x7fca1c06fc70 2023-10-12T22:30:43.257 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.115+0000 7fca1a59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 0x7fca1c10a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:43.258 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.116+0000 7fca1a59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 0x7fca1c10a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:49174/0 (socket says 172.21.15.131:49174) 2023-10-12T22:30:43.258 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.116+0000 7fca1a59c700 1 -- 172.21.15.131:0/3647913573 learned_addr learned my addr 172.21.15.131:0/3647913573 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:43.258 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.116+0000 7fca1a59c700 1 -- 172.21.15.131:0/3647913573 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c00b200 con 0x7fca1c06fc70 2023-10-12T22:30:43.258 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.116+0000 7fca1a59c700 1 --2- 172.21.15.131:0/3647913573 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 0x7fca1c10a9a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fca0c00e040 tx=0x7fca0c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:43.258 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.116+0000 7fca0affd700 1 -- 172.21.15.131:0/3647913573 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca0c01b410 con 0x7fca1c06fc70 2023-10-12T22:30:43.259 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.116+0000 7fca0affd700 1 -- 172.21.15.131:0/3647913573 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fca0c01b9e0 con 0x7fca1c06fc70 2023-10-12T22:30:43.259 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.116+0000 7fca20c95700 1 -- 172.21.15.131:0/3647913573 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1c10b090 con 0x7fca1c06fc70 2023-10-12T22:30:43.259 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.116+0000 7fca0affd700 1 -- 172.21.15.131:0/3647913573 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca0c01a770 con 0x7fca1c06fc70 2023-10-12T22:30:43.259 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.116+0000 7fca20c95700 1 -- 172.21.15.131:0/3647913573 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1c10b4a0 con 0x7fca1c06fc70 2023-10-12T22:30:43.259 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.117+0000 7fca0affd700 1 -- 172.21.15.131:0/3647913573 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fca0c00f070 con 0x7fca1c06fc70 2023-10-12T22:30:43.260 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.117+0000 7fca0affd700 1 --2- 172.21.15.131:0/3647913573 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7fca0002fc50 0x7fca00032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:43.260 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.117+0000 7fca0affd700 1 -- 172.21.15.131:0/3647913573 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fca0c040ee0 con 0x7fca1c06fc70 2023-10-12T22:30:43.260 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.117+0000 7fca19d9b700 1 --2- 172.21.15.131:0/3647913573 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7fca0002fc50 0x7fca00032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:43.260 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.117+0000 7fca20c95700 1 -- 172.21.15.131:0/3647913573 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1c045ad0 con 0x7fca1c06fc70 2023-10-12T22:30:43.260 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.117+0000 7fca19d9b700 1 --2- 172.21.15.131:0/3647913573 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7fca0002fc50 0x7fca00032110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fca100096a0 tx=0x7fca10006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:43.260 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.121+0000 7fca0affd700 1 -- 172.21.15.131:0/3647913573 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca0c016070 con 0x7fca1c06fc70 2023-10-12T22:30:43.314 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:43.313+0000 7fca20c95700 1 -- 172.21.15.131:0/3647913573 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fca1c0ffbf0 con 0x7fca1c06fc70 2023-10-12T22:30:43.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:43 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:42.209268+0000 mon.smithi131 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi131.efktma(active, since 1.00532s) 2023-10-12T22:30:43.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:43 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:42.435115+0000 mon.smithi131 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.131:0/621097872' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-12T22:30:44.209 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.207+0000 7fca0affd700 1 -- 172.21.15.131:0/3647913573 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fca0c010dd0 con 0x7fca1c06fc70 2023-10-12T22:30:44.210 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.209+0000 7fca0affd700 1 -- 172.21.15.131:0/3647913573 <== mon.0 v2:172.21.15.131:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7fca0c02ea40 con 0x7fca1c06fc70 2023-10-12T22:30:44.214 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.212+0000 7fca20c95700 1 -- 172.21.15.131:0/3647913573 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7fca0002fc50 msgr2=0x7fca00032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:44.214 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.212+0000 7fca20c95700 1 --2- 172.21.15.131:0/3647913573 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7fca0002fc50 0x7fca00032110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fca100096a0 tx=0x7fca10006b40).stop 2023-10-12T22:30:44.214 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.212+0000 7fca20c95700 1 -- 172.21.15.131:0/3647913573 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 msgr2=0x7fca1c10a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:44.217 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.212+0000 7fca20c95700 1 --2- 172.21.15.131:0/3647913573 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 0x7fca1c10a9a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fca0c00e040 tx=0x7fca0c006ee0).stop 2023-10-12T22:30:44.217 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.212+0000 7fca20c95700 1 -- 172.21.15.131:0/3647913573 shutdown_connections 2023-10-12T22:30:44.218 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.212+0000 7fca20c95700 1 --2- 172.21.15.131:0/3647913573 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7fca0002fc50 0x7fca00032110 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:44.218 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.212+0000 7fca20c95700 1 --2- 172.21.15.131:0/3647913573 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca1c06fc70 0x7fca1c10a9a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:44.219 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.212+0000 7fca20c95700 1 -- 172.21.15.131:0/3647913573 >> 172.21.15.131:0/3647913573 conn(0x7fca1c0f6280 msgr2=0x7fca1c0f7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:44.219 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.213+0000 7fca20c95700 1 -- 172.21.15.131:0/3647913573 shutdown_connections 2023-10-12T22:30:44.219 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.213+0000 7fca20c95700 1 -- 172.21.15.131:0/3647913573 wait complete. 2023-10-12T22:30:44.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:44 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:43.314825+0000 mon.smithi131 (mon.0) 24 : audit [INF] from='client.? 172.21.15.131:0/3647913573' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-10-12T22:30:44.592 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-10-12T22:30:44.920 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.917+0000 7f7fad552700 1 Processor -- start 2023-10-12T22:30:44.921 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.917+0000 7f7fad552700 1 -- start start 2023-10-12T22:30:44.921 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.917+0000 7f7fad552700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 0x7f7fa80faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:44.921 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.917+0000 7f7fad552700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa80fb480 con 0x7f7fa80faae0 2023-10-12T22:30:44.921 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.918+0000 7f7fa6ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 0x7f7fa80faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:44.922 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.918+0000 7f7fa6ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 0x7f7fa80faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:49200/0 (socket says 172.21.15.131:49200) 2023-10-12T22:30:44.922 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.918+0000 7f7fa6ffd700 1 -- 172.21.15.131:0/2164554039 learned_addr learned my addr 172.21.15.131:0/2164554039 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:44.922 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.918+0000 7f7fa6ffd700 1 -- 172.21.15.131:0/2164554039 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa80fb5c0 con 0x7f7fa80faae0 2023-10-12T22:30:44.922 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.918+0000 7f7fa6ffd700 1 --2- 172.21.15.131:0/2164554039 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 0x7f7fa80faf00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f7f9800b5f0 tx=0x7f7f98006ee0).ready entity=mon.0 client_cookie=32b976c29162930d server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:44.923 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.918+0000 7f7fa5ffb700 1 -- 172.21.15.131:0/2164554039 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f9800d600 con 0x7f7fa80faae0 2023-10-12T22:30:44.923 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.918+0000 7f7fa5ffb700 1 -- 172.21.15.131:0/2164554039 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7f9800dbd0 con 0x7f7fa80faae0 2023-10-12T22:30:44.923 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.918+0000 7f7fa5ffb700 1 -- 172.21.15.131:0/2164554039 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f98011c00 con 0x7f7fa80faae0 2023-10-12T22:30:44.923 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.919+0000 7f7fad552700 1 -- 172.21.15.131:0/2164554039 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 msgr2=0x7f7fa80faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:44.924 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.919+0000 7f7fad552700 1 --2- 172.21.15.131:0/2164554039 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 0x7f7fa80faf00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f7f9800b5f0 tx=0x7f7f98006ee0).stop 2023-10-12T22:30:44.924 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.919+0000 7f7fad552700 1 -- 172.21.15.131:0/2164554039 shutdown_connections 2023-10-12T22:30:44.924 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.919+0000 7f7fad552700 1 --2- 172.21.15.131:0/2164554039 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 0x7f7fa80faf00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:44.924 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.919+0000 7f7fad552700 1 -- 172.21.15.131:0/2164554039 >> 172.21.15.131:0/2164554039 conn(0x7f7fa80f6280 msgr2=0x7f7fa80f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:44.924 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.919+0000 7f7fad552700 1 -- 172.21.15.131:0/2164554039 shutdown_connections 2023-10-12T22:30:44.927 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.920+0000 7f7fad552700 1 -- 172.21.15.131:0/2164554039 wait complete. 2023-10-12T22:30:44.927 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.921+0000 7f7fad552700 1 Processor -- start 2023-10-12T22:30:44.928 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.921+0000 7f7fad552700 1 -- start start 2023-10-12T22:30:44.928 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.921+0000 7f7fad552700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 0x7f7fa810d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:44.928 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.921+0000 7f7fad552700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa806fcb0 con 0x7f7fa80faae0 2023-10-12T22:30:44.928 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.921+0000 7f7fa6ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 0x7f7fa810d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:44.928 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.921+0000 7f7fa6ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 0x7f7fa810d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:49202/0 (socket says 172.21.15.131:49202) 2023-10-12T22:30:44.929 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.921+0000 7f7fa6ffd700 1 -- 172.21.15.131:0/3823250080 learned_addr learned my addr 172.21.15.131:0/3823250080 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:44.929 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.921+0000 7f7fa6ffd700 1 -- 172.21.15.131:0/3823250080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9800b200 con 0x7f7fa80faae0 2023-10-12T22:30:44.929 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.921+0000 7f7fa6ffd700 1 --2- 172.21.15.131:0/3823250080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 0x7f7fa810d6f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f7f98002f40 tx=0x7f7f98006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:44.929 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.922+0000 7f7f8f7fe700 1 -- 172.21.15.131:0/3823250080 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f98011690 con 0x7f7fa80faae0 2023-10-12T22:30:44.929 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.922+0000 7f7f8f7fe700 1 -- 172.21.15.131:0/3823250080 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7f9801b880 con 0x7f7fa80faae0 2023-10-12T22:30:44.930 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.922+0000 7f7fad552700 1 -- 172.21.15.131:0/3823250080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa806feb0 con 0x7f7fa80faae0 2023-10-12T22:30:44.930 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.922+0000 7f7f8f7fe700 1 -- 172.21.15.131:0/3823250080 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f9801a6b0 con 0x7f7fa80faae0 2023-10-12T22:30:44.930 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.922+0000 7f7fad552700 1 -- 172.21.15.131:0/3823250080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa810dda0 con 0x7f7fa80faae0 2023-10-12T22:30:44.930 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.922+0000 7f7f8f7fe700 1 -- 172.21.15.131:0/3823250080 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f7f9800f070 con 0x7f7fa80faae0 2023-10-12T22:30:44.930 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.923+0000 7f7f8f7fe700 1 --2- 172.21.15.131:0/3823250080 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f7f9402fd50 0x7f7f94032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:44.931 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.923+0000 7f7fa67fc700 1 -- 172.21.15.131:0/3823250080 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f7f9402fd50 msgr2=0x7f7f94032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/1403842406 2023-10-12T22:30:44.931 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.923+0000 7f7fa67fc700 1 --2- 172.21.15.131:0/3823250080 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f7f9402fd50 0x7f7f94032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-12T22:30:44.931 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.923+0000 7f7f8f7fe700 1 -- 172.21.15.131:0/3823250080 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7f980255f0 con 0x7f7fa80faae0 2023-10-12T22:30:44.931 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.923+0000 7f7fad552700 1 -- 172.21.15.131:0/3823250080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa8045ad0 con 0x7f7fa80faae0 2023-10-12T22:30:44.932 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:44.928+0000 7f7f8f7fe700 1 -- 172.21.15.131:0/3823250080 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f98016070 con 0x7f7fa80faae0 2023-10-12T22:30:45.123 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.121+0000 7f7fad552700 1 -- 172.21.15.131:0/3823250080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7fa80fc9c0 con 0x7f7fa80faae0 2023-10-12T22:30:45.124 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.122+0000 7f7f8f7fe700 1 -- 172.21.15.131:0/3823250080 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7f7f98047060 con 0x7f7fa80faae0 2023-10-12T22:30:45.124 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2023-10-12T22:30:45.124 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 4, 2023-10-12T22:30:45.124 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "available": true, 2023-10-12T22:30:45.125 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "active_name": "smithi131.efktma", 2023-10-12T22:30:45.125 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_standby": 0 2023-10-12T22:30:45.125 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2023-10-12T22:30:45.127 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.124+0000 7f7fa67fc700 1 -- 172.21.15.131:0/3823250080 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f7f9402fd50 msgr2=0x7f7f94032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/1403842406 2023-10-12T22:30:45.127 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.124+0000 7f7fa67fc700 1 --2- 172.21.15.131:0/3823250080 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f7f9402fd50 0x7f7f94032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-12T22:30:45.127 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.124+0000 7f7fad552700 1 -- 172.21.15.131:0/3823250080 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f7f9402fd50 msgr2=0x7f7f94032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-12T22:30:45.127 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.124+0000 7f7fad552700 1 --2- 172.21.15.131:0/3823250080 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f7f9402fd50 0x7f7f94032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-12T22:30:45.127 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.124+0000 7f7fad552700 1 -- 172.21.15.131:0/3823250080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 msgr2=0x7f7fa810d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:45.128 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.124+0000 7f7fad552700 1 --2- 172.21.15.131:0/3823250080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 0x7f7fa810d6f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f7f98002f40 tx=0x7f7f98006ee0).stop 2023-10-12T22:30:45.128 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.124+0000 7f7fad552700 1 -- 172.21.15.131:0/3823250080 shutdown_connections 2023-10-12T22:30:45.128 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.124+0000 7f7fad552700 1 --2- 172.21.15.131:0/3823250080 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f7f9402fd50 0x7f7f94032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-12T22:30:45.128 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.124+0000 7f7fad552700 1 --2- 172.21.15.131:0/3823250080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fa80faae0 0x7f7fa810d6f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:45.128 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.124+0000 7f7fad552700 1 -- 172.21.15.131:0/3823250080 >> 172.21.15.131:0/3823250080 conn(0x7f7fa80f6280 msgr2=0x7f7fa80f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:45.129 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.124+0000 7f7fad552700 1 -- 172.21.15.131:0/3823250080 shutdown_connections 2023-10-12T22:30:45.129 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.124+0000 7f7fad552700 1 -- 172.21.15.131:0/3823250080 wait complete. 2023-10-12T22:30:45.450 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for the mgr to restart... 2023-10-12T22:30:45.450 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for mgr epoch 4... 2023-10-12T22:30:45.450 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-10-12T22:30:45.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:45 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:44.210269+0000 mon.smithi131 (mon.0) 25 : audit [INF] from='client.? 172.21.15.131:0/3647913573' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-10-12T22:30:45.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:45 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:44.210325+0000 mon.smithi131 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi131.efktma(active, since 3s) 2023-10-12T22:30:45.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:45 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:45.123264+0000 mon.smithi131 (mon.0) 27 : audit [DBG] from='client.? 172.21.15.131:0/3823250080' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-10-12T22:30:45.827 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.825+0000 7f7571dd0700 1 Processor -- start 2023-10-12T22:30:45.827 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.825+0000 7f7571dd0700 1 -- start start 2023-10-12T22:30:45.828 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.825+0000 7f7571dd0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 0x7f756c0fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:45.828 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.825+0000 7f7571dd0700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756c0fed60 con 0x7f756c06fc90 2023-10-12T22:30:45.828 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.826+0000 7f756b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 0x7f756c0fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:45.828 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.826+0000 7f756b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 0x7f756c0fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:49208/0 (socket says 172.21.15.131:49208) 2023-10-12T22:30:45.830 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.826+0000 7f756b7fe700 1 -- 172.21.15.131:0/1241835121 learned_addr learned my addr 172.21.15.131:0/1241835121 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:45.831 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.826+0000 7f756b7fe700 1 -- 172.21.15.131:0/1241835121 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c0feea0 con 0x7f756c06fc90 2023-10-12T22:30:45.831 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.826+0000 7f756b7fe700 1 --2- 172.21.15.131:0/1241835121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 0x7f756c0fe7e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f755c00b5f0 tx=0x7f755c006ee0).ready entity=mon.0 client_cookie=d7ccfac14594ee48 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:45.831 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.826+0000 7f756a7fc700 1 -- 172.21.15.131:0/1241835121 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f755c00d600 con 0x7f756c06fc90 2023-10-12T22:30:45.831 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.826+0000 7f756a7fc700 1 -- 172.21.15.131:0/1241835121 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f755c00dbd0 con 0x7f756c06fc90 2023-10-12T22:30:45.831 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.826+0000 7f756a7fc700 1 -- 172.21.15.131:0/1241835121 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f755c011c00 con 0x7f756c06fc90 2023-10-12T22:30:45.832 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.827+0000 7f7571dd0700 1 -- 172.21.15.131:0/1241835121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 msgr2=0x7f756c0fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:45.832 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.827+0000 7f7571dd0700 1 --2- 172.21.15.131:0/1241835121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 0x7f756c0fe7e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f755c00b5f0 tx=0x7f755c006ee0).stop 2023-10-12T22:30:45.832 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.827+0000 7f7571dd0700 1 -- 172.21.15.131:0/1241835121 shutdown_connections 2023-10-12T22:30:45.832 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.827+0000 7f7571dd0700 1 --2- 172.21.15.131:0/1241835121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 0x7f756c0fe7e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:45.832 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.827+0000 7f7571dd0700 1 -- 172.21.15.131:0/1241835121 >> 172.21.15.131:0/1241835121 conn(0x7f756c0f6280 msgr2=0x7f756c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:45.833 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.827+0000 7f7571dd0700 1 -- 172.21.15.131:0/1241835121 shutdown_connections 2023-10-12T22:30:45.833 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.827+0000 7f7571dd0700 1 -- 172.21.15.131:0/1241835121 wait complete. 2023-10-12T22:30:45.833 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.828+0000 7f7571dd0700 1 Processor -- start 2023-10-12T22:30:45.833 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.828+0000 7f7571dd0700 1 -- start start 2023-10-12T22:30:45.833 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.828+0000 7f7571dd0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 0x7f756c0fc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:45.833 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.828+0000 7f7571dd0700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756c0fa6e0 con 0x7f756c06fc90 2023-10-12T22:30:45.834 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.828+0000 7f756b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 0x7f756c0fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:45.834 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.828+0000 7f756b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 0x7f756c0fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:49222/0 (socket says 172.21.15.131:49222) 2023-10-12T22:30:45.834 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.828+0000 7f756b7fe700 1 -- 172.21.15.131:0/440454115 learned_addr learned my addr 172.21.15.131:0/440454115 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:45.834 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.829+0000 7f756b7fe700 1 -- 172.21.15.131:0/440454115 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c00b200 con 0x7f756c06fc90 2023-10-12T22:30:45.834 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.829+0000 7f756b7fe700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 0x7f756c0fc260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f755c00e040 tx=0x7f755c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:45.835 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.829+0000 7f7553fff700 1 -- 172.21.15.131:0/440454115 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f755c01b410 con 0x7f756c06fc90 2023-10-12T22:30:45.836 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.829+0000 7f7571dd0700 1 -- 172.21.15.131:0/440454115 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f756c0fa880 con 0x7f756c06fc90 2023-10-12T22:30:45.836 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.829+0000 7f7553fff700 1 -- 172.21.15.131:0/440454115 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f755c01b9e0 con 0x7f756c06fc90 2023-10-12T22:30:45.836 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.829+0000 7f7553fff700 1 -- 172.21.15.131:0/440454115 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f755c01a770 con 0x7f756c06fc90 2023-10-12T22:30:45.837 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.829+0000 7f7571dd0700 1 -- 172.21.15.131:0/440454115 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756c0fac90 con 0x7f756c06fc90 2023-10-12T22:30:45.837 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.830+0000 7f7553fff700 1 -- 172.21.15.131:0/440454115 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f755c00f070 con 0x7f756c06fc90 2023-10-12T22:30:45.837 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.830+0000 7f7553fff700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 0x7f7558032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:45.837 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.830+0000 7f7571dd0700 1 -- 172.21.15.131:0/440454115 --> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f756c059100 con 0x7f755802fd50 2023-10-12T22:30:45.837 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.830+0000 7f7553fff700 1 -- 172.21.15.131:0/440454115 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f755c041dc0 con 0x7f756c06fc90 2023-10-12T22:30:45.837 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.830+0000 7f756affd700 1 -- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 msgr2=0x7f7558032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/1403842406 2023-10-12T22:30:45.838 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:45.830+0000 7f756affd700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 0x7f7558032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-12T22:30:46.032 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:46.031+0000 7f756affd700 1 -- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 msgr2=0x7f7558032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/1403842406 2023-10-12T22:30:46.032 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:46.031+0000 7f756affd700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 0x7f7558032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-12T22:30:46.433 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:46.432+0000 7f756affd700 1 -- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 msgr2=0x7f7558032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/1403842406 2023-10-12T22:30:46.433 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:46.432+0000 7f756affd700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 0x7f7558032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-12T22:30:47.234 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:47.233+0000 7f756affd700 1 -- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 msgr2=0x7f7558032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/1403842406 2023-10-12T22:30:47.235 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:47.233+0000 7f756affd700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 0x7f7558032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-12T22:30:48.836 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:48.834+0000 7f756affd700 1 -- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 msgr2=0x7f7558032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/1403842406 2023-10-12T22:30:48.836 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:48.834+0000 7f756affd700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 0x7f7558032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-10-12T22:30:49.228 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:49.226+0000 7f7553fff700 1 -- 172.21.15.131:0/440454115 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f755c024790 con 0x7f756c06fc90 2023-10-12T22:30:49.228 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:49.227+0000 7f7553fff700 1 -- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 msgr2=0x7f7558032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-12T22:30:49.229 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:49.227+0000 7f7553fff700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 0x7f7558032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-12T22:30:49.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:49 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:49.176090+0000 mon.smithi131 (mon.0) 28 : cluster [INF] Active manager daemon smithi131.efktma restarted 2023-10-12T22:30:49.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:49 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:49.176468+0000 mon.smithi131 (mon.0) 29 : cluster [INF] Activating manager daemon smithi131.efktma 2023-10-12T22:30:49.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:49 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:49.178267+0000 mon.smithi131 (mon.0) 30 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-10-12T22:30:50.236 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.234+0000 7f7553fff700 1 -- 172.21.15.131:0/440454115 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f755c01bb50 con 0x7f756c06fc90 2023-10-12T22:30:50.236 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.234+0000 7f7553fff700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f7558033670 0x7f7558035a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:50.237 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.234+0000 7f7553fff700 1 -- 172.21.15.131:0/440454115 --> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f756c059100 con 0x7f7558033670 2023-10-12T22:30:50.237 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.235+0000 7f756affd700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f7558033670 0x7f7558035a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:50.238 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.235+0000 7f756affd700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f7558033670 0x7f7558035a60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f7560006fd0 tx=0x7f7560006cf0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:50.238 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.236+0000 7f7553fff700 1 -- 172.21.15.131:0/440454115 <== mgr.14114 v2:172.21.15.131:6800/3683890104 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f756c059100 con 0x7f7558033670 2023-10-12T22:30:50.244 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.243+0000 7f7571dd0700 1 -- 172.21.15.131:0/440454115 --> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f756c045ad0 con 0x7f7558033670 2023-10-12T22:30:50.245 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.243+0000 7f7553fff700 1 -- 172.21.15.131:0/440454115 <== mgr.14114 v2:172.21.15.131:6800/3683890104 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f756c045ad0 con 0x7f7558033670 2023-10-12T22:30:50.245 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2023-10-12T22:30:50.245 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2023-10-12T22:30:50.245 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "initialized": true 2023-10-12T22:30:50.246 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2023-10-12T22:30:50.247 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.243+0000 7f7571dd0700 1 -- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f7558033670 msgr2=0x7f7558035a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:50.247 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.243+0000 7f7571dd0700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f7558033670 0x7f7558035a60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f7560006fd0 tx=0x7f7560006cf0).stop 2023-10-12T22:30:50.247 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.243+0000 7f7571dd0700 1 -- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 msgr2=0x7f756c0fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:50.248 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.243+0000 7f7571dd0700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 0x7f756c0fc260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f755c00e040 tx=0x7f755c006ee0).stop 2023-10-12T22:30:50.248 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.244+0000 7f7571dd0700 1 -- 172.21.15.131:0/440454115 shutdown_connections 2023-10-12T22:30:50.248 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.244+0000 7f7571dd0700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f7558033670 0x7f7558035a60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:50.248 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.244+0000 7f7571dd0700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f756c06fc90 0x7f756c0fc260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:50.248 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.244+0000 7f7571dd0700 1 --2- 172.21.15.131:0/440454115 >> [v2:172.21.15.131:6800/1403842406,v1:172.21.15.131:6801/1403842406] conn(0x7f755802fd50 0x7f7558032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-12T22:30:50.249 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.244+0000 7f7571dd0700 1 -- 172.21.15.131:0/440454115 >> 172.21.15.131:0/440454115 conn(0x7f756c0f6280 msgr2=0x7f756c0f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:50.249 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.244+0000 7f7571dd0700 1 -- 172.21.15.131:0/440454115 shutdown_connections 2023-10-12T22:30:50.249 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:50.244+0000 7f7571dd0700 1 -- 172.21.15.131:0/440454115 wait complete. 2023-10-12T22:30:50.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:49.228682+0000 mon.smithi131 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi131.efktma(active, starting, since 0.0522823s) 2023-10-12T22:30:50.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.230807+0000 mon.smithi131 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2023-10-12T22:30:50.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.231390+0000 mon.smithi131 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mgr metadata", "who": "smithi131.efktma", "id": "smithi131.efktma"}]: dispatch 2023-10-12T22:30:50.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.232449+0000 mon.smithi131 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-12T22:30:50.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.232620+0000 mon.smithi131 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-12T22:30:50.491 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.234563+0000 mon.smithi131 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-12T22:30:50.491 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:49.236693+0000 mon.smithi131 (mon.0) 37 : cluster [INF] Manager daemon smithi131.efktma is now available 2023-10-12T22:30:50.491 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.252292+0000 mon.smithi131 (mon.0) 38 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:30:50.491 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.257668+0000 mon.smithi131 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:30:50.492 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.262924+0000 mon.smithi131 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:30:50.492 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.353711+0000 mon.smithi131 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:30:50.492 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.354429+0000 mon.smithi131 (mon.0) 42 : audit [DBG] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-12T22:30:50.492 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.355514+0000 mon.smithi131 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-12T22:30:50.492 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.357880+0000 mon.smithi131 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-12T22:30:50.492 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.367494+0000 mon.smithi131 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/mirror_snapshot_schedule"}]: dispatch 2023-10-12T22:30:50.493 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:49.368678+0000 mon.smithi131 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/trash_purge_schedule"}]: dispatch 2023-10-12T22:30:51.099 INFO:teuthology.orchestra.run.smithi131.stderr:mgr epoch 4 is available 2023-10-12T22:30:51.100 INFO:teuthology.orchestra.run.smithi131.stderr:Setting orchestrator backend to cephadm... 2023-10-12T22:30:51.100 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-10-12T22:30:51.435 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.429+0000 7f268a1ef700 1 Processor -- start 2023-10-12T22:30:51.436 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.429+0000 7f268a1ef700 1 -- start start 2023-10-12T22:30:51.436 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.429+0000 7f268a1ef700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 0x7f26840fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:51.436 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.429+0000 7f268a1ef700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26840fed60 con 0x7f268406fc90 2023-10-12T22:30:51.436 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.430+0000 7f26837fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 0x7f26840fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:51.436 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.430+0000 7f26837fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 0x7f26840fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:41792/0 (socket says 172.21.15.131:41792) 2023-10-12T22:30:51.437 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.430+0000 7f26837fe700 1 -- 172.21.15.131:0/1163100648 learned_addr learned my addr 172.21.15.131:0/1163100648 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:51.437 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.430+0000 7f26837fe700 1 -- 172.21.15.131:0/1163100648 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26840feea0 con 0x7f268406fc90 2023-10-12T22:30:51.437 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.430+0000 7f26837fe700 1 --2- 172.21.15.131:0/1163100648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 0x7f26840fe7e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f267800b5f0 tx=0x7f2678006ee0).ready entity=mon.0 client_cookie=7407dd4d9086602a server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:51.437 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.430+0000 7f26827fc700 1 -- 172.21.15.131:0/1163100648 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f267800d600 con 0x7f268406fc90 2023-10-12T22:30:51.437 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.430+0000 7f26827fc700 1 -- 172.21.15.131:0/1163100648 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f267800dbd0 con 0x7f268406fc90 2023-10-12T22:30:51.438 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.430+0000 7f26827fc700 1 -- 172.21.15.131:0/1163100648 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2678011c00 con 0x7f268406fc90 2023-10-12T22:30:51.438 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.431+0000 7f268a1ef700 1 -- 172.21.15.131:0/1163100648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 msgr2=0x7f26840fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:51.438 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.431+0000 7f268a1ef700 1 --2- 172.21.15.131:0/1163100648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 0x7f26840fe7e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f267800b5f0 tx=0x7f2678006ee0).stop 2023-10-12T22:30:51.438 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.431+0000 7f268a1ef700 1 -- 172.21.15.131:0/1163100648 shutdown_connections 2023-10-12T22:30:51.438 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.431+0000 7f268a1ef700 1 --2- 172.21.15.131:0/1163100648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 0x7f26840fe7e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:51.439 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.431+0000 7f268a1ef700 1 -- 172.21.15.131:0/1163100648 >> 172.21.15.131:0/1163100648 conn(0x7f26840f6280 msgr2=0x7f26840f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:51.439 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.431+0000 7f268a1ef700 1 -- 172.21.15.131:0/1163100648 shutdown_connections 2023-10-12T22:30:51.439 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.431+0000 7f268a1ef700 1 -- 172.21.15.131:0/1163100648 wait complete. 2023-10-12T22:30:51.439 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.431+0000 7f268a1ef700 1 Processor -- start 2023-10-12T22:30:51.439 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.432+0000 7f268a1ef700 1 -- start start 2023-10-12T22:30:51.439 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.432+0000 7f268a1ef700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 0x7f268410a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:51.440 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.432+0000 7f268a1ef700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268410ae90 con 0x7f268406fc90 2023-10-12T22:30:51.440 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.432+0000 7f26837fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 0x7f268410a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:51.440 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.432+0000 7f26837fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 0x7f268410a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:41798/0 (socket says 172.21.15.131:41798) 2023-10-12T22:30:51.440 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.432+0000 7f26837fe700 1 -- 172.21.15.131:0/641705534 learned_addr learned my addr 172.21.15.131:0/641705534 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:51.440 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.432+0000 7f26837fe700 1 -- 172.21.15.131:0/641705534 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267800b200 con 0x7f268406fc90 2023-10-12T22:30:51.441 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.432+0000 7f26837fe700 1 --2- 172.21.15.131:0/641705534 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 0x7f268410a9a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f267800e040 tx=0x7f2678006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:51.441 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.432+0000 7f266bfff700 1 -- 172.21.15.131:0/641705534 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f267801b410 con 0x7f268406fc90 2023-10-12T22:30:51.441 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.433+0000 7f266bfff700 1 -- 172.21.15.131:0/641705534 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f267801b9e0 con 0x7f268406fc90 2023-10-12T22:30:51.441 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.433+0000 7f268a1ef700 1 -- 172.21.15.131:0/641705534 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268410b090 con 0x7f268406fc90 2023-10-12T22:30:51.441 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.433+0000 7f266bfff700 1 -- 172.21.15.131:0/641705534 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f267801a770 con 0x7f268406fc90 2023-10-12T22:30:51.442 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.433+0000 7f268a1ef700 1 -- 172.21.15.131:0/641705534 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268410b4a0 con 0x7f268406fc90 2023-10-12T22:30:51.442 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.433+0000 7f266bfff700 1 -- 172.21.15.131:0/641705534 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f267800f070 con 0x7f268406fc90 2023-10-12T22:30:51.442 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.433+0000 7f266bfff700 1 --2- 172.21.15.131:0/641705534 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f267002fd50 0x7f2670032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:51.443 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.434+0000 7f266bfff700 1 -- 172.21.15.131:0/641705534 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2678041e60 con 0x7f268406fc90 2023-10-12T22:30:51.443 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.434+0000 7f268a1ef700 1 -- 172.21.15.131:0/641705534 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2684045ad0 con 0x7f268406fc90 2023-10-12T22:30:51.443 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.434+0000 7f2682ffd700 1 --2- 172.21.15.131:0/641705534 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f267002fd50 0x7f2670032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:51.443 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.434+0000 7f2682ffd700 1 --2- 172.21.15.131:0/641705534 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f267002fd50 0x7f2670032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f26740096a0 tx=0x7f2674006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:51.444 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.438+0000 7f266bfff700 1 -- 172.21.15.131:0/641705534 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2678016030 con 0x7f268406fc90 2023-10-12T22:30:51.618 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.617+0000 7f268a1ef700 1 -- 172.21.15.131:0/641705534 --> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f2684104890 con 0x7f267002fd50 2023-10-12T22:30:51.623 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.621+0000 7f266bfff700 1 -- 172.21.15.131:0/641705534 <== mgr.14114 v2:172.21.15.131:6800/3683890104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2684104890 con 0x7f267002fd50 2023-10-12T22:30:51.626 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.624+0000 7f268a1ef700 1 -- 172.21.15.131:0/641705534 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f267002fd50 msgr2=0x7f2670032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:51.626 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.624+0000 7f268a1ef700 1 --2- 172.21.15.131:0/641705534 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f267002fd50 0x7f2670032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f26740096a0 tx=0x7f2674006b40).stop 2023-10-12T22:30:51.626 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.624+0000 7f268a1ef700 1 -- 172.21.15.131:0/641705534 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 msgr2=0x7f268410a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:51.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.624+0000 7f268a1ef700 1 --2- 172.21.15.131:0/641705534 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 0x7f268410a9a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f267800e040 tx=0x7f2678006ee0).stop 2023-10-12T22:30:51.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.624+0000 7f268a1ef700 1 -- 172.21.15.131:0/641705534 shutdown_connections 2023-10-12T22:30:51.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.624+0000 7f268a1ef700 1 --2- 172.21.15.131:0/641705534 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f267002fd50 0x7f2670032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:51.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.624+0000 7f268a1ef700 1 --2- 172.21.15.131:0/641705534 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268406fc90 0x7f268410a9a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:51.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.624+0000 7f268a1ef700 1 -- 172.21.15.131:0/641705534 >> 172.21.15.131:0/641705534 conn(0x7f26840f6280 msgr2=0x7f26840f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:51.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.624+0000 7f268a1ef700 1 -- 172.21.15.131:0/641705534 shutdown_connections 2023-10-12T22:30:51.628 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:51.624+0000 7f268a1ef700 1 -- 172.21.15.131:0/641705534 wait complete. 2023-10-12T22:30:51.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:51 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:50.237148+0000 mgr.smithi131.efktma (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-10-12T22:30:51.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:51 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:50.237947+0000 mon.smithi131 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi131.efktma(active, since 1.06154s) 2023-10-12T22:30:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:51 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:50.244588+0000 mgr.smithi131.efktma (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-10-12T22:30:51.966 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-10-12T22:30:52.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.299+0000 7ff40cbf3700 1 Processor -- start 2023-10-12T22:30:52.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.299+0000 7ff40cbf3700 1 -- start start 2023-10-12T22:30:52.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.299+0000 7ff40cbf3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 0x7ff408100c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:52.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.299+0000 7ff40cbf3700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4080fc6c0 con 0x7ff40806fc30 2023-10-12T22:30:52.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.300+0000 7ff40659c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 0x7ff408100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:52.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.300+0000 7ff40659c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 0x7ff408100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:41806/0 (socket says 172.21.15.131:41806) 2023-10-12T22:30:52.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.300+0000 7ff40659c700 1 -- 172.21.15.131:0/4224890522 learned_addr learned my addr 172.21.15.131:0/4224890522 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:52.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.300+0000 7ff40659c700 1 -- 172.21.15.131:0/4224890522 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff408101130 con 0x7ff40806fc30 2023-10-12T22:30:52.304 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.300+0000 7ff40659c700 1 --2- 172.21.15.131:0/4224890522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 0x7ff408100c40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ff3fc00b5f0 tx=0x7ff3fc006ee0).ready entity=mon.0 client_cookie=10a28afc7c1bb534 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:52.304 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.300+0000 7ff40559a700 1 -- 172.21.15.131:0/4224890522 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3fc00d600 con 0x7ff40806fc30 2023-10-12T22:30:52.304 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.300+0000 7ff40559a700 1 -- 172.21.15.131:0/4224890522 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff3fc00dbd0 con 0x7ff40806fc30 2023-10-12T22:30:52.304 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.300+0000 7ff40559a700 1 -- 172.21.15.131:0/4224890522 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3fc011c00 con 0x7ff40806fc30 2023-10-12T22:30:52.304 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.301+0000 7ff40cbf3700 1 -- 172.21.15.131:0/4224890522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 msgr2=0x7ff408100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:52.305 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.301+0000 7ff40cbf3700 1 --2- 172.21.15.131:0/4224890522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 0x7ff408100c40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ff3fc00b5f0 tx=0x7ff3fc006ee0).stop 2023-10-12T22:30:52.306 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.301+0000 7ff40cbf3700 1 -- 172.21.15.131:0/4224890522 shutdown_connections 2023-10-12T22:30:52.307 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.301+0000 7ff40cbf3700 1 --2- 172.21.15.131:0/4224890522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 0x7ff408100c40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:52.307 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.301+0000 7ff40cbf3700 1 -- 172.21.15.131:0/4224890522 >> 172.21.15.131:0/4224890522 conn(0x7ff4080f6280 msgr2=0x7ff4080f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:52.307 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.301+0000 7ff40cbf3700 1 -- 172.21.15.131:0/4224890522 shutdown_connections 2023-10-12T22:30:52.307 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.301+0000 7ff40cbf3700 1 -- 172.21.15.131:0/4224890522 wait complete. 2023-10-12T22:30:52.307 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.301+0000 7ff40cbf3700 1 Processor -- start 2023-10-12T22:30:52.308 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.301+0000 7ff40cbf3700 1 -- start start 2023-10-12T22:30:52.308 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.302+0000 7ff40cbf3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 0x7ff40810cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:52.308 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.302+0000 7ff40cbf3700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40810d090 con 0x7ff40806fc30 2023-10-12T22:30:52.308 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.302+0000 7ff40659c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 0x7ff40810cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:52.308 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.302+0000 7ff40659c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 0x7ff40810cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:41820/0 (socket says 172.21.15.131:41820) 2023-10-12T22:30:52.309 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.302+0000 7ff40659c700 1 -- 172.21.15.131:0/2200885056 learned_addr learned my addr 172.21.15.131:0/2200885056 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:52.309 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.302+0000 7ff40659c700 1 -- 172.21.15.131:0/2200885056 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc00b200 con 0x7ff40806fc30 2023-10-12T22:30:52.309 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.302+0000 7ff40659c700 1 --2- 172.21.15.131:0/2200885056 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 0x7ff40810cba0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff3fc00e040 tx=0x7ff3fc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:52.309 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.302+0000 7ff3f6ffd700 1 -- 172.21.15.131:0/2200885056 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3fc01b410 con 0x7ff40806fc30 2023-10-12T22:30:52.309 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.302+0000 7ff3f6ffd700 1 -- 172.21.15.131:0/2200885056 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff3fc01b9e0 con 0x7ff40806fc30 2023-10-12T22:30:52.310 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.302+0000 7ff3f6ffd700 1 -- 172.21.15.131:0/2200885056 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3fc01a770 con 0x7ff40806fc30 2023-10-12T22:30:52.310 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.302+0000 7ff40cbf3700 1 -- 172.21.15.131:0/2200885056 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40810d290 con 0x7ff40806fc30 2023-10-12T22:30:52.310 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.302+0000 7ff40cbf3700 1 -- 172.21.15.131:0/2200885056 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40810d6a0 con 0x7ff40806fc30 2023-10-12T22:30:52.310 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.303+0000 7ff3f6ffd700 1 -- 172.21.15.131:0/2200885056 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff3fc00f070 con 0x7ff40806fc30 2023-10-12T22:30:52.310 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.303+0000 7ff3f6ffd700 1 --2- 172.21.15.131:0/2200885056 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7ff3ec02fdb0 0x7ff3ec032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:52.310 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.303+0000 7ff3f6ffd700 1 -- 172.21.15.131:0/2200885056 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff3fc040f70 con 0x7ff40806fc30 2023-10-12T22:30:52.311 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.303+0000 7ff405d9b700 1 --2- 172.21.15.131:0/2200885056 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7ff3ec02fdb0 0x7ff3ec032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:52.311 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.303+0000 7ff40cbf3700 1 -- 172.21.15.131:0/2200885056 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff408059100 con 0x7ff40806fc30 2023-10-12T22:30:52.311 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.304+0000 7ff405d9b700 1 --2- 172.21.15.131:0/2200885056 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7ff3ec02fdb0 0x7ff3ec032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff3f80096a0 tx=0x7ff3f8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:52.312 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.307+0000 7ff3f6ffd700 1 -- 172.21.15.131:0/2200885056 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3fc016020 con 0x7ff40806fc30 2023-10-12T22:30:52.461 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.459+0000 7ff40cbf3700 1 -- 172.21.15.131:0/2200885056 --> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7ff408106ac0 con 0x7ff3ec02fdb0 2023-10-12T22:30:52.461 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.460+0000 7ff3f6ffd700 1 -- 172.21.15.131:0/2200885056 <== mgr.14114 v2:172.21.15.131:6800/3683890104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7ff408106ac0 con 0x7ff3ec02fdb0 2023-10-12T22:30:52.462 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: value unchanged 2023-10-12T22:30:52.463 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.462+0000 7ff40cbf3700 1 -- 172.21.15.131:0/2200885056 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7ff3ec02fdb0 msgr2=0x7ff3ec032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:52.464 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.462+0000 7ff40cbf3700 1 --2- 172.21.15.131:0/2200885056 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7ff3ec02fdb0 0x7ff3ec032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff3f80096a0 tx=0x7ff3f8006b40).stop 2023-10-12T22:30:52.464 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.462+0000 7ff40cbf3700 1 -- 172.21.15.131:0/2200885056 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 msgr2=0x7ff40810cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:52.464 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.462+0000 7ff40cbf3700 1 --2- 172.21.15.131:0/2200885056 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 0x7ff40810cba0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff3fc00e040 tx=0x7ff3fc006ee0).stop 2023-10-12T22:30:52.464 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.462+0000 7ff40cbf3700 1 -- 172.21.15.131:0/2200885056 shutdown_connections 2023-10-12T22:30:52.464 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.462+0000 7ff40cbf3700 1 --2- 172.21.15.131:0/2200885056 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7ff3ec02fdb0 0x7ff3ec032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:52.465 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.462+0000 7ff40cbf3700 1 --2- 172.21.15.131:0/2200885056 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff40806fc30 0x7ff40810cba0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:52.465 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.462+0000 7ff40cbf3700 1 -- 172.21.15.131:0/2200885056 >> 172.21.15.131:0/2200885056 conn(0x7ff4080f6280 msgr2=0x7ff4080f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:52.465 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.462+0000 7ff40cbf3700 1 -- 172.21.15.131:0/2200885056 shutdown_connections 2023-10-12T22:30:52.465 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:52.462+0000 7ff40cbf3700 1 -- 172.21.15.131:0/2200885056 wait complete. 2023-10-12T22:30:52.622 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:30:51.243950+0000 mon.smithi131 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi131.efktma(active, since 2s) 2023-10-12T22:30:52.623 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:51.622742+0000 mon.smithi131 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:30:52.623 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:51.626454+0000 mon.smithi131 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-12T22:30:52.807 INFO:teuthology.orchestra.run.smithi131.stderr:Generating ssh key... 2023-10-12T22:30:52.807 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-10-12T22:30:53.136 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.133+0000 7f82a27b1700 1 Processor -- start 2023-10-12T22:30:53.136 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.133+0000 7f82a27b1700 1 -- start start 2023-10-12T22:30:53.136 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.133+0000 7f82a27b1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 0x7f829c0faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:53.137 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.133+0000 7f82a27b1700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0fb480 con 0x7f829c0faae0 2023-10-12T22:30:53.137 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.133+0000 7f829bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 0x7f829c0faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:53.137 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.133+0000 7f829bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 0x7f829c0faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:41826/0 (socket says 172.21.15.131:41826) 2023-10-12T22:30:53.137 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.133+0000 7f829bfff700 1 -- 172.21.15.131:0/947357519 learned_addr learned my addr 172.21.15.131:0/947357519 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:53.137 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.133+0000 7f829bfff700 1 -- 172.21.15.131:0/947357519 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c0fb5c0 con 0x7f829c0faae0 2023-10-12T22:30:53.138 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.133+0000 7f829bfff700 1 --2- 172.21.15.131:0/947357519 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 0x7f829c0faf00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f828c00b5f0 tx=0x7f828c006ee0).ready entity=mon.0 client_cookie=a95422f7003bcbd server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:53.138 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.134+0000 7f829affd700 1 -- 172.21.15.131:0/947357519 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f828c00d600 con 0x7f829c0faae0 2023-10-12T22:30:53.138 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.134+0000 7f829affd700 1 -- 172.21.15.131:0/947357519 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f828c00dbd0 con 0x7f829c0faae0 2023-10-12T22:30:53.138 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.134+0000 7f829affd700 1 -- 172.21.15.131:0/947357519 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f828c011c00 con 0x7f829c0faae0 2023-10-12T22:30:53.138 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.134+0000 7f82a27b1700 1 -- 172.21.15.131:0/947357519 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 msgr2=0x7f829c0faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:53.138 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.134+0000 7f82a27b1700 1 --2- 172.21.15.131:0/947357519 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 0x7f829c0faf00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f828c00b5f0 tx=0x7f828c006ee0).stop 2023-10-12T22:30:53.139 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.134+0000 7f82a27b1700 1 -- 172.21.15.131:0/947357519 shutdown_connections 2023-10-12T22:30:53.139 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.134+0000 7f82a27b1700 1 --2- 172.21.15.131:0/947357519 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 0x7f829c0faf00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:53.139 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.135+0000 7f82a27b1700 1 -- 172.21.15.131:0/947357519 >> 172.21.15.131:0/947357519 conn(0x7f829c0f6280 msgr2=0x7f829c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:53.139 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.135+0000 7f82a27b1700 1 -- 172.21.15.131:0/947357519 shutdown_connections 2023-10-12T22:30:53.139 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.135+0000 7f82a27b1700 1 -- 172.21.15.131:0/947357519 wait complete. 2023-10-12T22:30:53.141 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.135+0000 7f82a27b1700 1 Processor -- start 2023-10-12T22:30:53.141 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.136+0000 7f82a27b1700 1 -- start start 2023-10-12T22:30:53.142 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.136+0000 7f82a27b1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 0x7f829c1098d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:53.142 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.136+0000 7f82a27b1700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c109dc0 con 0x7f829c0faae0 2023-10-12T22:30:53.142 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.136+0000 7f829bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 0x7f829c1098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:53.142 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.136+0000 7f829bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 0x7f829c1098d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:41832/0 (socket says 172.21.15.131:41832) 2023-10-12T22:30:53.142 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.136+0000 7f829bfff700 1 -- 172.21.15.131:0/3864198840 learned_addr learned my addr 172.21.15.131:0/3864198840 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:53.143 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.136+0000 7f829bfff700 1 -- 172.21.15.131:0/3864198840 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c00b200 con 0x7f829c0faae0 2023-10-12T22:30:53.143 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.136+0000 7f829bfff700 1 --2- 172.21.15.131:0/3864198840 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 0x7f829c1098d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f828c00b1d0 tx=0x7f828c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:53.143 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.136+0000 7f8298ff9700 1 -- 172.21.15.131:0/3864198840 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f828c011a20 con 0x7f829c0faae0 2023-10-12T22:30:53.143 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.136+0000 7f82a27b1700 1 -- 172.21.15.131:0/3864198840 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829c109fc0 con 0x7f829c0faae0 2023-10-12T22:30:53.143 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.136+0000 7f8298ff9700 1 -- 172.21.15.131:0/3864198840 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f828c01b880 con 0x7f829c0faae0 2023-10-12T22:30:53.144 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.136+0000 7f8298ff9700 1 -- 172.21.15.131:0/3864198840 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f828c01a700 con 0x7f829c0faae0 2023-10-12T22:30:53.144 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.137+0000 7f82a27b1700 1 -- 172.21.15.131:0/3864198840 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829c10a3d0 con 0x7f829c0faae0 2023-10-12T22:30:53.144 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.137+0000 7f82a27b1700 1 -- 172.21.15.131:0/3864198840 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829c045ad0 con 0x7f829c0faae0 2023-10-12T22:30:53.144 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.137+0000 7f8298ff9700 1 -- 172.21.15.131:0/3864198840 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f828c00f070 con 0x7f829c0faae0 2023-10-12T22:30:53.144 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.138+0000 7f8298ff9700 1 --2- 172.21.15.131:0/3864198840 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f828802fd50 0x7f8288032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:53.145 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.138+0000 7f8298ff9700 1 -- 172.21.15.131:0/3864198840 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f828c041190 con 0x7f829c0faae0 2023-10-12T22:30:53.145 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.138+0000 7f829b7fe700 1 --2- 172.21.15.131:0/3864198840 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f828802fd50 0x7f8288032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:53.145 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.138+0000 7f829b7fe700 1 --2- 172.21.15.131:0/3864198840 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f828802fd50 0x7f8288032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f82900096a0 tx=0x7f8290006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:53.145 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.142+0000 7f8298ff9700 1 -- 172.21.15.131:0/3864198840 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f828c022310 con 0x7f829c0faae0 2023-10-12T22:30:53.296 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.295+0000 7f82a27b1700 1 -- 172.21.15.131:0/3864198840 --> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f829c10ad10 con 0x7f828802fd50 2023-10-12T22:30:53.465 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.464+0000 7f8298ff9700 1 -- 172.21.15.131:0/3864198840 <== mgr.14114 v2:172.21.15.131:6800/3683890104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f829c10ad10 con 0x7f828802fd50 2023-10-12T22:30:53.468 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.466+0000 7f82a27b1700 1 -- 172.21.15.131:0/3864198840 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f828802fd50 msgr2=0x7f8288032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:53.468 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.466+0000 7f82a27b1700 1 --2- 172.21.15.131:0/3864198840 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f828802fd50 0x7f8288032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f82900096a0 tx=0x7f8290006b40).stop 2023-10-12T22:30:53.468 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.466+0000 7f82a27b1700 1 -- 172.21.15.131:0/3864198840 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 msgr2=0x7f829c1098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:53.468 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.466+0000 7f82a27b1700 1 --2- 172.21.15.131:0/3864198840 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 0x7f829c1098d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f828c00b1d0 tx=0x7f828c006ee0).stop 2023-10-12T22:30:53.468 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.466+0000 7f82a27b1700 1 -- 172.21.15.131:0/3864198840 shutdown_connections 2023-10-12T22:30:53.469 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.466+0000 7f82a27b1700 1 --2- 172.21.15.131:0/3864198840 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f828802fd50 0x7f8288032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:53.469 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.466+0000 7f82a27b1700 1 --2- 172.21.15.131:0/3864198840 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f829c0faae0 0x7f829c1098d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:53.469 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.466+0000 7f82a27b1700 1 -- 172.21.15.131:0/3864198840 >> 172.21.15.131:0/3864198840 conn(0x7f829c0f6280 msgr2=0x7f829c0f7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:53.469 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.466+0000 7f82a27b1700 1 -- 172.21.15.131:0/3864198840 shutdown_connections 2023-10-12T22:30:53.469 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:53.466+0000 7f82a27b1700 1 -- 172.21.15.131:0/3864198840 wait complete. 2023-10-12T22:30:53.631 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:53 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:51.618997+0000 mgr.smithi131.efktma (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:30:53.798 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-10-12T22:30:54.122 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.119+0000 7fe5856b6700 1 Processor -- start 2023-10-12T22:30:54.122 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.119+0000 7fe5856b6700 1 -- start start 2023-10-12T22:30:54.122 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.120+0000 7fe5856b6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 0x7fe5800faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:54.123 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.120+0000 7fe5856b6700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800fb3e0 con 0x7fe5800faad0 2023-10-12T22:30:54.123 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.120+0000 7fe57effd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 0x7fe5800faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:54.123 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.120+0000 7fe57effd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 0x7fe5800faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:41838/0 (socket says 172.21.15.131:41838) 2023-10-12T22:30:54.123 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.120+0000 7fe57effd700 1 -- 172.21.15.131:0/2630129825 learned_addr learned my addr 172.21.15.131:0/2630129825 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:54.123 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.120+0000 7fe57effd700 1 -- 172.21.15.131:0/2630129825 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5800fb520 con 0x7fe5800faad0 2023-10-12T22:30:54.124 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.120+0000 7fe57effd700 1 --2- 172.21.15.131:0/2630129825 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 0x7fe5800faef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe57400b5f0 tx=0x7fe574006ee0).ready entity=mon.0 client_cookie=9d60c3a102165c83 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:54.124 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.121+0000 7fe57dffb700 1 -- 172.21.15.131:0/2630129825 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe57400d600 con 0x7fe5800faad0 2023-10-12T22:30:54.126 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.121+0000 7fe57dffb700 1 -- 172.21.15.131:0/2630129825 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe57400dbd0 con 0x7fe5800faad0 2023-10-12T22:30:54.126 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.121+0000 7fe57dffb700 1 -- 172.21.15.131:0/2630129825 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe574011c00 con 0x7fe5800faad0 2023-10-12T22:30:54.127 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.121+0000 7fe5856b6700 1 -- 172.21.15.131:0/2630129825 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 msgr2=0x7fe5800faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:54.127 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.121+0000 7fe5856b6700 1 --2- 172.21.15.131:0/2630129825 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 0x7fe5800faef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe57400b5f0 tx=0x7fe574006ee0).stop 2023-10-12T22:30:54.127 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.121+0000 7fe5856b6700 1 -- 172.21.15.131:0/2630129825 shutdown_connections 2023-10-12T22:30:54.127 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.121+0000 7fe5856b6700 1 --2- 172.21.15.131:0/2630129825 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 0x7fe5800faef0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:54.127 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.121+0000 7fe5856b6700 1 -- 172.21.15.131:0/2630129825 >> 172.21.15.131:0/2630129825 conn(0x7fe5800f6310 msgr2=0x7fe5800f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:54.128 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.121+0000 7fe5856b6700 1 -- 172.21.15.131:0/2630129825 shutdown_connections 2023-10-12T22:30:54.128 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.121+0000 7fe5856b6700 1 -- 172.21.15.131:0/2630129825 wait complete. 2023-10-12T22:30:54.128 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.122+0000 7fe5856b6700 1 Processor -- start 2023-10-12T22:30:54.128 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.122+0000 7fe5856b6700 1 -- start start 2023-10-12T22:30:54.128 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.122+0000 7fe5856b6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 0x7fe58010cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:54.128 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.122+0000 7fe5856b6700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58010d040 con 0x7fe5800faad0 2023-10-12T22:30:54.129 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.122+0000 7fe57effd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 0x7fe58010cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:54.129 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.122+0000 7fe57effd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 0x7fe58010cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:41844/0 (socket says 172.21.15.131:41844) 2023-10-12T22:30:54.129 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.122+0000 7fe57effd700 1 -- 172.21.15.131:0/3220820729 learned_addr learned my addr 172.21.15.131:0/3220820729 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:54.129 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.122+0000 7fe57effd700 1 -- 172.21.15.131:0/3220820729 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57400b200 con 0x7fe5800faad0 2023-10-12T22:30:54.129 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.122+0000 7fe57effd700 1 --2- 172.21.15.131:0/3220820729 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 0x7fe58010cb50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fe57400b1d0 tx=0x7fe574006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:54.130 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.123+0000 7fe5677fe700 1 -- 172.21.15.131:0/3220820729 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe57401b410 con 0x7fe5800faad0 2023-10-12T22:30:54.130 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.123+0000 7fe5856b6700 1 -- 172.21.15.131:0/3220820729 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58010d240 con 0x7fe5800faad0 2023-10-12T22:30:54.130 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.123+0000 7fe5677fe700 1 -- 172.21.15.131:0/3220820729 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe57401b9e0 con 0x7fe5800faad0 2023-10-12T22:30:54.130 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.123+0000 7fe5677fe700 1 -- 172.21.15.131:0/3220820729 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe57401a840 con 0x7fe5800faad0 2023-10-12T22:30:54.130 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.123+0000 7fe5856b6700 1 -- 172.21.15.131:0/3220820729 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58010d650 con 0x7fe5800faad0 2023-10-12T22:30:54.131 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.123+0000 7fe5677fe700 1 -- 172.21.15.131:0/3220820729 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe57400f070 con 0x7fe5800faad0 2023-10-12T22:30:54.131 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.124+0000 7fe5677fe700 1 --2- 172.21.15.131:0/3220820729 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe56c02fd50 0x7fe56c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:54.131 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.124+0000 7fe5677fe700 1 -- 172.21.15.131:0/3220820729 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe574042230 con 0x7fe5800faad0 2023-10-12T22:30:54.131 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.124+0000 7fe57e7fc700 1 --2- 172.21.15.131:0/3220820729 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe56c02fd50 0x7fe56c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:54.131 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.124+0000 7fe5856b6700 1 -- 172.21.15.131:0/3220820729 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe580059100 con 0x7fe5800faad0 2023-10-12T22:30:54.132 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.124+0000 7fe57e7fc700 1 --2- 172.21.15.131:0/3220820729 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe56c02fd50 0x7fe56c032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe5700096a0 tx=0x7fe570006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:54.132 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.128+0000 7fe5677fe700 1 -- 172.21.15.131:0/3220820729 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe574016070 con 0x7fe5800faad0 2023-10-12T22:30:54.281 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.279+0000 7fe5856b6700 1 -- 172.21.15.131:0/3220820729 --> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe580106cf0 con 0x7fe56c02fd50 2023-10-12T22:30:54.281 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.280+0000 7fe5677fe700 1 -- 172.21.15.131:0/3220820729 <== mgr.14114 v2:172.21.15.131:6800/3683890104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fe580106cf0 con 0x7fe56c02fd50 2023-10-12T22:30:54.281 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlCssUDxr+d0RkbRblv9bheg4OhGeOyDhYM/fUdjLsOyCDPL61VVfpFoUG2eLOR8dEJOZiiU+3lfh16ELIs1b5Ex4by18zEjGNWYDtWjeBE2lF1ACqR2cJ9luZUJZWOXpf3xadXi3Npbmb0m4yq7MS/LuvoQtFr6pir3dJWAmG3er1g7oQsijdJLLlKIpe67DHiydNxo1Mpi4TDIGJ/MsvkJiRrUb03ZjnOsNDXvJ3pgD5+rrPY0+iBsP59ifYCMzmz/HDL7yrpndYfycPhAQKojZQ2drrdsfxU+vlAqa4BeeQ3y0xpN+D//jXuILLvUj2IFJ61rqOUzDvMKQAPswr/LqNNDk8jkz4l2Kh9OaTPyl90LA1KDYAvNHatV1bm0/vP++xgEcuH3mzpe6Wb4K/bzM3zcs3gm9RKdtTNT27nmv4OAZ5AcNtvGJxHI06a2Gmd+G7UCZvuQnXyWe94ZkBvf0BMZ7z6yPsCCnxkuFNQ1uBR+tg6RD3xPZkwuNS36M= ceph-e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:30:54.284 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.282+0000 7fe5856b6700 1 -- 172.21.15.131:0/3220820729 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe56c02fd50 msgr2=0x7fe56c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:54.284 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.282+0000 7fe5856b6700 1 --2- 172.21.15.131:0/3220820729 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe56c02fd50 0x7fe56c032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe5700096a0 tx=0x7fe570006b40).stop 2023-10-12T22:30:54.284 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.282+0000 7fe5856b6700 1 -- 172.21.15.131:0/3220820729 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 msgr2=0x7fe58010cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:54.284 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.282+0000 7fe5856b6700 1 --2- 172.21.15.131:0/3220820729 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 0x7fe58010cb50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fe57400b1d0 tx=0x7fe574006ee0).stop 2023-10-12T22:30:54.284 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.282+0000 7fe5856b6700 1 -- 172.21.15.131:0/3220820729 shutdown_connections 2023-10-12T22:30:54.285 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.282+0000 7fe5856b6700 1 --2- 172.21.15.131:0/3220820729 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe56c02fd50 0x7fe56c032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:54.285 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.282+0000 7fe5856b6700 1 --2- 172.21.15.131:0/3220820729 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5800faad0 0x7fe58010cb50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:54.285 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.282+0000 7fe5856b6700 1 -- 172.21.15.131:0/3220820729 >> 172.21.15.131:0/3220820729 conn(0x7fe5800f6310 msgr2=0x7fe58006f830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:54.285 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.282+0000 7fe5856b6700 1 -- 172.21.15.131:0/3220820729 shutdown_connections 2023-10-12T22:30:54.285 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.282+0000 7fe5856b6700 1 -- 172.21.15.131:0/3220820729 wait complete. 2023-10-12T22:30:54.542 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:52.461397+0000 mgr.smithi131.efktma (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:30:54.543 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:53.462939+0000 mon.smithi131 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:30:54.543 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:53.464650+0000 mon.smithi131 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:30:54.625 INFO:teuthology.orchestra.run.smithi131.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-10-12T22:30:54.625 INFO:teuthology.orchestra.run.smithi131.stderr:Adding key to root@localhost authorized_keys... 2023-10-12T22:30:54.626 INFO:teuthology.orchestra.run.smithi131.stderr:Adding host smithi131... 2023-10-12T22:30:54.627 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi131 172.21.15.131 2023-10-12T22:30:54.945 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.942+0000 7fd81d525700 1 Processor -- start 2023-10-12T22:30:54.946 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.942+0000 7fd81d525700 1 -- start start 2023-10-12T22:30:54.946 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.942+0000 7fd81d525700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 0x7fd8180faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:54.946 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.942+0000 7fd81d525700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8180fb480 con 0x7fd8180faae0 2023-10-12T22:30:54.946 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.942+0000 7fd816ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 0x7fd8180faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:54.947 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.942+0000 7fd816ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 0x7fd8180faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:41846/0 (socket says 172.21.15.131:41846) 2023-10-12T22:30:54.947 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.942+0000 7fd816ffd700 1 -- 172.21.15.131:0/3415285747 learned_addr learned my addr 172.21.15.131:0/3415285747 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:54.947 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.943+0000 7fd816ffd700 1 -- 172.21.15.131:0/3415285747 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8180fb5c0 con 0x7fd8180faae0 2023-10-12T22:30:54.947 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.943+0000 7fd816ffd700 1 --2- 172.21.15.131:0/3415285747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 0x7fd8180faf00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fd80c00b5f0 tx=0x7fd80c006ee0).ready entity=mon.0 client_cookie=9b8b93b7e2f24648 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:54.947 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.943+0000 7fd815ffb700 1 -- 172.21.15.131:0/3415285747 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd80c00d600 con 0x7fd8180faae0 2023-10-12T22:30:54.948 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.943+0000 7fd815ffb700 1 -- 172.21.15.131:0/3415285747 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd80c00dbd0 con 0x7fd8180faae0 2023-10-12T22:30:54.948 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.943+0000 7fd815ffb700 1 -- 172.21.15.131:0/3415285747 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd80c011c00 con 0x7fd8180faae0 2023-10-12T22:30:54.948 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.944+0000 7fd81d525700 1 -- 172.21.15.131:0/3415285747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 msgr2=0x7fd8180faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:54.948 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.944+0000 7fd81d525700 1 --2- 172.21.15.131:0/3415285747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 0x7fd8180faf00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fd80c00b5f0 tx=0x7fd80c006ee0).stop 2023-10-12T22:30:54.948 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.944+0000 7fd81d525700 1 -- 172.21.15.131:0/3415285747 shutdown_connections 2023-10-12T22:30:54.949 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.944+0000 7fd81d525700 1 --2- 172.21.15.131:0/3415285747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 0x7fd8180faf00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:54.949 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.944+0000 7fd81d525700 1 -- 172.21.15.131:0/3415285747 >> 172.21.15.131:0/3415285747 conn(0x7fd8180f6280 msgr2=0x7fd8180f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:54.949 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.944+0000 7fd81d525700 1 -- 172.21.15.131:0/3415285747 shutdown_connections 2023-10-12T22:30:54.949 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.944+0000 7fd81d525700 1 -- 172.21.15.131:0/3415285747 wait complete. 2023-10-12T22:30:54.949 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.944+0000 7fd81d525700 1 Processor -- start 2023-10-12T22:30:54.950 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.944+0000 7fd81d525700 1 -- start start 2023-10-12T22:30:54.951 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.945+0000 7fd81d525700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 0x7fd818100df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:54.951 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.945+0000 7fd81d525700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8180ff270 con 0x7fd8180faae0 2023-10-12T22:30:54.952 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.945+0000 7fd816ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 0x7fd818100df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:54.952 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.945+0000 7fd816ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 0x7fd818100df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:41854/0 (socket says 172.21.15.131:41854) 2023-10-12T22:30:54.952 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.945+0000 7fd816ffd700 1 -- 172.21.15.131:0/392508198 learned_addr learned my addr 172.21.15.131:0/392508198 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:54.952 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.945+0000 7fd816ffd700 1 -- 172.21.15.131:0/392508198 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c00b200 con 0x7fd8180faae0 2023-10-12T22:30:54.952 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.945+0000 7fd816ffd700 1 --2- 172.21.15.131:0/392508198 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 0x7fd818100df0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd80c00b1d0 tx=0x7fd80c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:54.953 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.945+0000 7fd7ff7fe700 1 -- 172.21.15.131:0/392508198 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd80c011a20 con 0x7fd8180faae0 2023-10-12T22:30:54.953 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.945+0000 7fd7ff7fe700 1 -- 172.21.15.131:0/392508198 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd80c01b880 con 0x7fd8180faae0 2023-10-12T22:30:54.953 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.945+0000 7fd81d525700 1 -- 172.21.15.131:0/392508198 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8180ff410 con 0x7fd8180faae0 2023-10-12T22:30:54.953 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.945+0000 7fd7ff7fe700 1 -- 172.21.15.131:0/392508198 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd80c01a700 con 0x7fd8180faae0 2023-10-12T22:30:54.953 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.945+0000 7fd81d525700 1 -- 172.21.15.131:0/392508198 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8180ff820 con 0x7fd8180faae0 2023-10-12T22:30:54.954 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.946+0000 7fd7ff7fe700 1 -- 172.21.15.131:0/392508198 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd80c00f070 con 0x7fd8180faae0 2023-10-12T22:30:54.954 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.946+0000 7fd7ff7fe700 1 --2- 172.21.15.131:0/392508198 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fd80402fd50 0x7fd804032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:54.954 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.946+0000 7fd7ff7fe700 1 -- 172.21.15.131:0/392508198 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd80c0410d0 con 0x7fd8180faae0 2023-10-12T22:30:54.954 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.946+0000 7fd8167fc700 1 --2- 172.21.15.131:0/392508198 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fd80402fd50 0x7fd804032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:54.954 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.946+0000 7fd81d525700 1 -- 172.21.15.131:0/392508198 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd818059100 con 0x7fd8180faae0 2023-10-12T22:30:54.955 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.947+0000 7fd8167fc700 1 --2- 172.21.15.131:0/392508198 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fd80402fd50 0x7fd804032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd8080096a0 tx=0x7fd808006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:54.955 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:54.951+0000 7fd7ff7fe700 1 -- 172.21.15.131:0/392508198 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd80c016070 con 0x7fd8180faae0 2023-10-12T22:30:55.123 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:55.122+0000 7fd81d525700 1 -- 172.21.15.131:0/392508198 --> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi131", "addr": "172.21.15.131", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8180fc930 con 0x7fd80402fd50 2023-10-12T22:30:55.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:55 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:30:53.297030+0000 mgr.smithi131.efktma (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2023-10-12T22:30:55.859 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Added host 'smithi131' 2023-10-12T22:30:55.859 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:55.856+0000 7fd7ff7fe700 1 -- 172.21.15.131:0/392508198 <== mgr.14114 v2:172.21.15.131:6800/3683890104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fd8180fc930 con 0x7fd80402fd50 2023-10-12T22:30:55.862 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:55.859+0000 7fd81d525700 1 -- 172.21.15.131:0/392508198 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fd80402fd50 msgr2=0x7fd804032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:55.862 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:55.859+0000 7fd81d525700 1 --2- 172.21.15.131:0/392508198 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fd80402fd50 0x7fd804032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd8080096a0 tx=0x7fd808006b40).stop 2023-10-12T22:30:55.863 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:55.859+0000 7fd81d525700 1 -- 172.21.15.131:0/392508198 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 msgr2=0x7fd818100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:55.863 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:55.859+0000 7fd81d525700 1 --2- 172.21.15.131:0/392508198 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 0x7fd818100df0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd80c00b1d0 tx=0x7fd80c006ee0).stop 2023-10-12T22:30:55.863 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:55.859+0000 7fd81d525700 1 -- 172.21.15.131:0/392508198 shutdown_connections 2023-10-12T22:30:55.864 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:55.859+0000 7fd81d525700 1 --2- 172.21.15.131:0/392508198 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fd80402fd50 0x7fd804032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:55.864 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:55.859+0000 7fd81d525700 1 --2- 172.21.15.131:0/392508198 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd8180faae0 0x7fd818100df0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:55.864 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:55.859+0000 7fd81d525700 1 -- 172.21.15.131:0/392508198 >> 172.21.15.131:0/392508198 conn(0x7fd8180f6280 msgr2=0x7fd81806f890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:55.865 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:55.859+0000 7fd81d525700 1 -- 172.21.15.131:0/392508198 shutdown_connections 2023-10-12T22:30:55.865 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:55.859+0000 7fd81d525700 1 -- 172.21.15.131:0/392508198 wait complete. 2023-10-12T22:30:56.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:54.281319+0000 mgr.smithi131.efktma (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:30:56.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:55.123926+0000 mgr.smithi131.efktma (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi131", "addr": "172.21.15.131", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:30:56.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:55.856831+0000 mon.smithi131 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:30:56.741 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:55.857354+0000 mon.smithi131 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-12T22:30:57.483 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying mon service with default placement... 2023-10-12T22:30:57.483 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-10-12T22:30:57.485 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:57 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:30:55.857106+0000 mgr.smithi131.efktma (mgr.14114) 9 : cephadm [INF] Added host smithi131 2023-10-12T22:30:58.849 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.670+0000 7fb8b5f52700 1 Processor -- start 2023-10-12T22:30:58.850 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.670+0000 7fb8b5f52700 1 -- start start 2023-10-12T22:30:58.850 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.671+0000 7fb8b5f52700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 0x7fb8b00fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:58.850 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.671+0000 7fb8b5f52700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b00fed20 con 0x7fb8b006fcd0 2023-10-12T22:30:58.850 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.671+0000 7fb8af7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 0x7fb8b00fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:58.850 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.671+0000 7fb8af7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 0x7fb8b00fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:41858/0 (socket says 172.21.15.131:41858) 2023-10-12T22:30:58.851 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.671+0000 7fb8af7fe700 1 -- 172.21.15.131:0/1457791283 learned_addr learned my addr 172.21.15.131:0/1457791283 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:58.851 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.671+0000 7fb8af7fe700 1 -- 172.21.15.131:0/1457791283 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b00fee60 con 0x7fb8b006fcd0 2023-10-12T22:30:58.851 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.671+0000 7fb8af7fe700 1 --2- 172.21.15.131:0/1457791283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 0x7fb8b00fe7a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb89800b5f0 tx=0x7fb898006ee0).ready entity=mon.0 client_cookie=2aca5dc1ec7c9fd server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:58.851 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.672+0000 7fb8ae7fc700 1 -- 172.21.15.131:0/1457791283 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb89800d600 con 0x7fb8b006fcd0 2023-10-12T22:30:58.851 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.672+0000 7fb8ae7fc700 1 -- 172.21.15.131:0/1457791283 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb89800dbd0 con 0x7fb8b006fcd0 2023-10-12T22:30:58.852 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.672+0000 7fb8ae7fc700 1 -- 172.21.15.131:0/1457791283 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb898011c00 con 0x7fb8b006fcd0 2023-10-12T22:30:58.852 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.672+0000 7fb8b5f52700 1 -- 172.21.15.131:0/1457791283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 msgr2=0x7fb8b00fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:58.852 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.672+0000 7fb8b5f52700 1 --2- 172.21.15.131:0/1457791283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 0x7fb8b00fe7a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb89800b5f0 tx=0x7fb898006ee0).stop 2023-10-12T22:30:58.852 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.672+0000 7fb8b5f52700 1 -- 172.21.15.131:0/1457791283 shutdown_connections 2023-10-12T22:30:58.852 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.672+0000 7fb8b5f52700 1 --2- 172.21.15.131:0/1457791283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 0x7fb8b00fe7a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:58.852 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.672+0000 7fb8b5f52700 1 -- 172.21.15.131:0/1457791283 >> 172.21.15.131:0/1457791283 conn(0x7fb8b00f6280 msgr2=0x7fb8b00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:58.853 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.672+0000 7fb8b5f52700 1 -- 172.21.15.131:0/1457791283 shutdown_connections 2023-10-12T22:30:58.853 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.672+0000 7fb8b5f52700 1 -- 172.21.15.131:0/1457791283 wait complete. 2023-10-12T22:30:58.853 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.673+0000 7fb8b5f52700 1 Processor -- start 2023-10-12T22:30:58.853 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.673+0000 7fb8b5f52700 1 -- start start 2023-10-12T22:30:58.853 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.673+0000 7fb8b5f52700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 0x7fb8b010a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:58.853 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.673+0000 7fb8b5f52700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b010ae90 con 0x7fb8b006fcd0 2023-10-12T22:30:58.854 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.674+0000 7fb8af7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 0x7fb8b010a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:58.854 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.674+0000 7fb8af7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 0x7fb8b010a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:41874/0 (socket says 172.21.15.131:41874) 2023-10-12T22:30:58.854 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.674+0000 7fb8af7fe700 1 -- 172.21.15.131:0/3378239216 learned_addr learned my addr 172.21.15.131:0/3378239216 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:30:58.854 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.674+0000 7fb8af7fe700 1 -- 172.21.15.131:0/3378239216 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89800b200 con 0x7fb8b006fcd0 2023-10-12T22:30:58.854 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.674+0000 7fb8af7fe700 1 --2- 172.21.15.131:0/3378239216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 0x7fb8b010a9a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb89800e040 tx=0x7fb898006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:58.855 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.674+0000 7fb8a7fff700 1 -- 172.21.15.131:0/3378239216 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb89801b410 con 0x7fb8b006fcd0 2023-10-12T22:30:58.855 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.674+0000 7fb8a7fff700 1 -- 172.21.15.131:0/3378239216 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb89801b9e0 con 0x7fb8b006fcd0 2023-10-12T22:30:58.855 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.674+0000 7fb8b5f52700 1 -- 172.21.15.131:0/3378239216 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b010b090 con 0x7fb8b006fcd0 2023-10-12T22:30:58.855 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.674+0000 7fb8a7fff700 1 -- 172.21.15.131:0/3378239216 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb89801a770 con 0x7fb8b006fcd0 2023-10-12T22:30:58.855 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.674+0000 7fb8b5f52700 1 -- 172.21.15.131:0/3378239216 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b010b4a0 con 0x7fb8b006fcd0 2023-10-12T22:30:58.855 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.675+0000 7fb8a7fff700 1 -- 172.21.15.131:0/3378239216 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb89800f070 con 0x7fb8b006fcd0 2023-10-12T22:30:58.856 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.675+0000 7fb8a7fff700 1 --2- 172.21.15.131:0/3378239216 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fb89002fdb0 0x7fb890032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:30:58.856 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.675+0000 7fb8a7fff700 1 -- 172.21.15.131:0/3378239216 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb898040f30 con 0x7fb8b006fcd0 2023-10-12T22:30:58.856 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.675+0000 7fb8b5f52700 1 -- 172.21.15.131:0/3378239216 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b0045ad0 con 0x7fb8b006fcd0 2023-10-12T22:30:58.856 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.675+0000 7fb8aeffd700 1 --2- 172.21.15.131:0/3378239216 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fb89002fdb0 0x7fb890032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:30:58.856 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.675+0000 7fb8aeffd700 1 --2- 172.21.15.131:0/3378239216 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fb89002fdb0 0x7fb890032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb8a00096a0 tx=0x7fb8a0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:30:58.856 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.679+0000 7fb8a7fff700 1 -- 172.21.15.131:0/3378239216 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb898016070 con 0x7fb8b006fcd0 2023-10-12T22:30:58.857 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.856+0000 7fb8b5f52700 1 -- 172.21.15.131:0/3378239216 --> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8b0104930 con 0x7fb89002fdb0 2023-10-12T22:30:58.861 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.860+0000 7fb8a7fff700 1 -- 172.21.15.131:0/3378239216 <== mgr.14114 v2:172.21.15.131:6800/3683890104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb8b0104930 con 0x7fb89002fdb0 2023-10-12T22:30:58.862 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled mon update... 2023-10-12T22:30:58.864 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.862+0000 7fb8b5f52700 1 -- 172.21.15.131:0/3378239216 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fb89002fdb0 msgr2=0x7fb890032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:58.864 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.862+0000 7fb8b5f52700 1 --2- 172.21.15.131:0/3378239216 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fb89002fdb0 0x7fb890032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb8a00096a0 tx=0x7fb8a0006b40).stop 2023-10-12T22:30:58.864 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.862+0000 7fb8b5f52700 1 -- 172.21.15.131:0/3378239216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 msgr2=0x7fb8b010a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:30:58.865 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.862+0000 7fb8b5f52700 1 --2- 172.21.15.131:0/3378239216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 0x7fb8b010a9a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb89800e040 tx=0x7fb898006ee0).stop 2023-10-12T22:30:58.865 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.862+0000 7fb8b5f52700 1 -- 172.21.15.131:0/3378239216 shutdown_connections 2023-10-12T22:30:58.865 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.862+0000 7fb8b5f52700 1 --2- 172.21.15.131:0/3378239216 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fb89002fdb0 0x7fb890032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:58.865 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.862+0000 7fb8b5f52700 1 --2- 172.21.15.131:0/3378239216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b006fcd0 0x7fb8b010a9a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:30:58.865 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.862+0000 7fb8b5f52700 1 -- 172.21.15.131:0/3378239216 >> 172.21.15.131:0/3378239216 conn(0x7fb8b00f6280 msgr2=0x7fb8b00f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:30:58.866 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.862+0000 7fb8b5f52700 1 -- 172.21.15.131:0/3378239216 shutdown_connections 2023-10-12T22:30:58.866 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:30:58.863+0000 7fb8b5f52700 1 -- 172.21.15.131:0/3378239216 wait complete. 2023-10-12T22:31:00.241 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:59 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:58.858506+0000 mgr.smithi131.efktma (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:31:00.242 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:59 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:30:58.859234+0000 mgr.smithi131.efktma (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-10-12T22:31:00.242 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:30:59 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:30:58.861656+0000 mon.smithi131 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:31:00.752 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying mgr service with default placement... 2023-10-12T22:31:00.752 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-10-12T22:31:02.098 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.921+0000 7f37df846700 1 Processor -- start 2023-10-12T22:31:02.098 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.921+0000 7f37df846700 1 -- start start 2023-10-12T22:31:02.099 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.921+0000 7f37df846700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 0x7f37d80fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:02.099 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.922+0000 7f37df846700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d80fed20 con 0x7f37d806fcd0 2023-10-12T22:31:02.099 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.922+0000 7f37dd5e2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 0x7f37d80fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:02.099 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.922+0000 7f37dd5e2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 0x7f37d80fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48152/0 (socket says 172.21.15.131:48152) 2023-10-12T22:31:02.099 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.922+0000 7f37dd5e2700 1 -- 172.21.15.131:0/2932679062 learned_addr learned my addr 172.21.15.131:0/2932679062 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:02.100 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.922+0000 7f37dd5e2700 1 -- 172.21.15.131:0/2932679062 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d80fee60 con 0x7f37d806fcd0 2023-10-12T22:31:02.100 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.922+0000 7f37dd5e2700 1 --2- 172.21.15.131:0/2932679062 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 0x7f37d80fe7a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f37c800b5f0 tx=0x7f37c8006ee0).ready entity=mon.0 client_cookie=55a48c1e75457ac3 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:02.100 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.922+0000 7f37c7fff700 1 -- 172.21.15.131:0/2932679062 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37c800d600 con 0x7f37d806fcd0 2023-10-12T22:31:02.100 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.922+0000 7f37c7fff700 1 -- 172.21.15.131:0/2932679062 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f37c800dbd0 con 0x7f37d806fcd0 2023-10-12T22:31:02.100 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.923+0000 7f37c7fff700 1 -- 172.21.15.131:0/2932679062 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37c8011c00 con 0x7f37d806fcd0 2023-10-12T22:31:02.101 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.923+0000 7f37df846700 1 -- 172.21.15.131:0/2932679062 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 msgr2=0x7f37d80fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:02.101 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.923+0000 7f37df846700 1 --2- 172.21.15.131:0/2932679062 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 0x7f37d80fe7a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f37c800b5f0 tx=0x7f37c8006ee0).stop 2023-10-12T22:31:02.101 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.923+0000 7f37df846700 1 -- 172.21.15.131:0/2932679062 shutdown_connections 2023-10-12T22:31:02.101 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.923+0000 7f37df846700 1 --2- 172.21.15.131:0/2932679062 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 0x7f37d80fe7a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:02.101 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.923+0000 7f37df846700 1 -- 172.21.15.131:0/2932679062 >> 172.21.15.131:0/2932679062 conn(0x7f37d80f6280 msgr2=0x7f37d80f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:02.102 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.923+0000 7f37df846700 1 -- 172.21.15.131:0/2932679062 shutdown_connections 2023-10-12T22:31:02.102 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.923+0000 7f37df846700 1 -- 172.21.15.131:0/2932679062 wait complete. 2023-10-12T22:31:02.102 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.924+0000 7f37df846700 1 Processor -- start 2023-10-12T22:31:02.102 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.924+0000 7f37df846700 1 -- start start 2023-10-12T22:31:02.102 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.924+0000 7f37df846700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 0x7f37d80fc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:02.103 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.924+0000 7f37df846700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d80fa6a0 con 0x7f37d806fcd0 2023-10-12T22:31:02.103 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.924+0000 7f37dd5e2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 0x7f37d80fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:02.103 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.924+0000 7f37dd5e2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 0x7f37d80fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48160/0 (socket says 172.21.15.131:48160) 2023-10-12T22:31:02.103 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.924+0000 7f37dd5e2700 1 -- 172.21.15.131:0/1611856914 learned_addr learned my addr 172.21.15.131:0/1611856914 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:02.104 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.924+0000 7f37dd5e2700 1 -- 172.21.15.131:0/1611856914 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c800b200 con 0x7f37d806fcd0 2023-10-12T22:31:02.104 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.924+0000 7f37dd5e2700 1 --2- 172.21.15.131:0/1611856914 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 0x7f37d80fc220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f37c800e040 tx=0x7f37c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:02.104 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.925+0000 7f37c5ffb700 1 -- 172.21.15.131:0/1611856914 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37c801b410 con 0x7f37d806fcd0 2023-10-12T22:31:02.105 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.925+0000 7f37c5ffb700 1 -- 172.21.15.131:0/1611856914 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f37c801b9e0 con 0x7f37d806fcd0 2023-10-12T22:31:02.105 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.925+0000 7f37c5ffb700 1 -- 172.21.15.131:0/1611856914 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37c801a770 con 0x7f37d806fcd0 2023-10-12T22:31:02.105 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.925+0000 7f37df846700 1 -- 172.21.15.131:0/1611856914 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d80fa840 con 0x7f37d806fcd0 2023-10-12T22:31:02.106 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.925+0000 7f37df846700 1 -- 172.21.15.131:0/1611856914 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d80fac50 con 0x7f37d806fcd0 2023-10-12T22:31:02.106 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.926+0000 7f37c5ffb700 1 -- 172.21.15.131:0/1611856914 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f37c800f070 con 0x7f37d806fcd0 2023-10-12T22:31:02.106 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.926+0000 7f37c5ffb700 1 --2- 172.21.15.131:0/1611856914 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f37c002fdb0 0x7f37c0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:02.107 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.926+0000 7f37c5ffb700 1 -- 172.21.15.131:0/1611856914 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f37c8040f40 con 0x7f37d806fcd0 2023-10-12T22:31:02.107 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.926+0000 7f37df846700 1 -- 172.21.15.131:0/1611856914 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d8045ad0 con 0x7f37d806fcd0 2023-10-12T22:31:02.107 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.926+0000 7f37dcde1700 1 --2- 172.21.15.131:0/1611856914 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f37c002fdb0 0x7f37c0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:02.108 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.926+0000 7f37dcde1700 1 --2- 172.21.15.131:0/1611856914 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f37c002fdb0 0x7f37c0032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f37d40096a0 tx=0x7f37d4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:02.108 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:01.930+0000 7f37c5ffb700 1 -- 172.21.15.131:0/1611856914 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37c8016080 con 0x7f37d806fcd0 2023-10-12T22:31:02.109 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:02.107+0000 7f37df846700 1 -- 172.21.15.131:0/1611856914 --> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f37d8104860 con 0x7f37c002fdb0 2023-10-12T22:31:02.112 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:02.110+0000 7f37c5ffb700 1 -- 172.21.15.131:0/1611856914 <== mgr.14114 v2:172.21.15.131:6800/3683890104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f37d8104860 con 0x7f37c002fdb0 2023-10-12T22:31:02.112 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-10-12T22:31:02.114 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:02.112+0000 7f37df846700 1 -- 172.21.15.131:0/1611856914 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f37c002fdb0 msgr2=0x7f37c0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:02.115 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:02.112+0000 7f37df846700 1 --2- 172.21.15.131:0/1611856914 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f37c002fdb0 0x7f37c0032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f37d40096a0 tx=0x7f37d4006b40).stop 2023-10-12T22:31:02.115 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:02.112+0000 7f37df846700 1 -- 172.21.15.131:0/1611856914 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 msgr2=0x7f37d80fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:02.115 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:02.112+0000 7f37df846700 1 --2- 172.21.15.131:0/1611856914 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 0x7f37d80fc220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f37c800e040 tx=0x7f37c8006ee0).stop 2023-10-12T22:31:02.115 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:02.112+0000 7f37df846700 1 -- 172.21.15.131:0/1611856914 shutdown_connections 2023-10-12T22:31:02.115 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:02.112+0000 7f37df846700 1 --2- 172.21.15.131:0/1611856914 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f37c002fdb0 0x7f37c0032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:02.116 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:02.112+0000 7f37df846700 1 --2- 172.21.15.131:0/1611856914 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37d806fcd0 0x7f37d80fc220 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:02.116 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:02.112+0000 7f37df846700 1 -- 172.21.15.131:0/1611856914 >> 172.21.15.131:0/1611856914 conn(0x7f37d80f6280 msgr2=0x7f37d80f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:02.116 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:02.112+0000 7f37df846700 1 -- 172.21.15.131:0/1611856914 shutdown_connections 2023-10-12T22:31:02.116 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:02.112+0000 7f37df846700 1 -- 172.21.15.131:0/1611856914 wait complete. 2023-10-12T22:31:02.998 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying crash service with default placement... 2023-10-12T22:31:02.998 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-10-12T22:31:03.463 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.460+0000 7fc9c1c99700 1 Processor -- start 2023-10-12T22:31:03.463 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.460+0000 7fc9c1c99700 1 -- start start 2023-10-12T22:31:03.463 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.460+0000 7fc9c1c99700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 0x7fc9bc0fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:03.464 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.460+0000 7fc9c1c99700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc0fed20 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.464 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.461+0000 7fc9bb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 0x7fc9bc0fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:03.464 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.461+0000 7fc9bb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 0x7fc9bc0fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48162/0 (socket says 172.21.15.131:48162) 2023-10-12T22:31:03.464 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.461+0000 7fc9bb7fe700 1 -- 172.21.15.131:0/3058440865 learned_addr learned my addr 172.21.15.131:0/3058440865 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:03.464 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.461+0000 7fc9bb7fe700 1 -- 172.21.15.131:0/3058440865 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc0fee60 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.465 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.461+0000 7fc9bb7fe700 1 --2- 172.21.15.131:0/3058440865 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 0x7fc9bc0fe7a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc9ac00b5f0 tx=0x7fc9ac006ee0).ready entity=mon.0 client_cookie=cda9f946edb4c6f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:03.465 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.461+0000 7fc9ba7fc700 1 -- 172.21.15.131:0/3058440865 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9ac00d600 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.465 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.461+0000 7fc9ba7fc700 1 -- 172.21.15.131:0/3058440865 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc9ac00dbd0 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.465 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.461+0000 7fc9ba7fc700 1 -- 172.21.15.131:0/3058440865 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9ac011c00 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.468 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.462+0000 7fc9c1c99700 1 -- 172.21.15.131:0/3058440865 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 msgr2=0x7fc9bc0fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:03.468 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.462+0000 7fc9c1c99700 1 --2- 172.21.15.131:0/3058440865 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 0x7fc9bc0fe7a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc9ac00b5f0 tx=0x7fc9ac006ee0).stop 2023-10-12T22:31:03.468 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.462+0000 7fc9c1c99700 1 -- 172.21.15.131:0/3058440865 shutdown_connections 2023-10-12T22:31:03.468 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.462+0000 7fc9c1c99700 1 --2- 172.21.15.131:0/3058440865 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 0x7fc9bc0fe7a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:03.468 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.462+0000 7fc9c1c99700 1 -- 172.21.15.131:0/3058440865 >> 172.21.15.131:0/3058440865 conn(0x7fc9bc0f6280 msgr2=0x7fc9bc0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:03.469 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.462+0000 7fc9c1c99700 1 -- 172.21.15.131:0/3058440865 shutdown_connections 2023-10-12T22:31:03.469 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.462+0000 7fc9c1c99700 1 -- 172.21.15.131:0/3058440865 wait complete. 2023-10-12T22:31:03.469 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.463+0000 7fc9c1c99700 1 Processor -- start 2023-10-12T22:31:03.469 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.463+0000 7fc9c1c99700 1 -- start start 2023-10-12T22:31:03.469 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.463+0000 7fc9c1c99700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 0x7fc9bc10a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:03.470 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.464+0000 7fc9c1c99700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc10aee0 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.470 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.464+0000 7fc9bb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 0x7fc9bc10a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:03.470 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.464+0000 7fc9bb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 0x7fc9bc10a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48166/0 (socket says 172.21.15.131:48166) 2023-10-12T22:31:03.470 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.464+0000 7fc9bb7fe700 1 -- 172.21.15.131:0/1886996664 learned_addr learned my addr 172.21.15.131:0/1886996664 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:03.470 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.464+0000 7fc9bb7fe700 1 -- 172.21.15.131:0/1886996664 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac00b200 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.471 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.464+0000 7fc9bb7fe700 1 --2- 172.21.15.131:0/1886996664 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 0x7fc9bc10a9f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc9ac000c00 tx=0x7fc9ac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:03.471 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.464+0000 7fc9a3fff700 1 -- 172.21.15.131:0/1886996664 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9ac011690 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.471 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.464+0000 7fc9a3fff700 1 -- 172.21.15.131:0/1886996664 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc9ac00d9f0 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.471 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.464+0000 7fc9c1c99700 1 -- 172.21.15.131:0/1886996664 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc10b0e0 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.471 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.464+0000 7fc9a3fff700 1 -- 172.21.15.131:0/1886996664 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9ac01b710 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.472 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.464+0000 7fc9c1c99700 1 -- 172.21.15.131:0/1886996664 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9bc10b4f0 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.472 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.465+0000 7fc9a3fff700 1 -- 172.21.15.131:0/1886996664 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc9ac00f070 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.472 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.465+0000 7fc9c1c99700 1 -- 172.21.15.131:0/1886996664 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9bc045ad0 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.472 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.465+0000 7fc9a3fff700 1 --2- 172.21.15.131:0/1886996664 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fc9a802fdb0 0x7fc9a8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:03.472 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.465+0000 7fc9a3fff700 1 -- 172.21.15.131:0/1886996664 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc9ac041140 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.473 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.466+0000 7fc9baffd700 1 --2- 172.21.15.131:0/1886996664 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fc9a802fdb0 0x7fc9a8032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:03.473 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.466+0000 7fc9baffd700 1 --2- 172.21.15.131:0/1886996664 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fc9a802fdb0 0x7fc9a8032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc9b00096a0 tx=0x7fc9b0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:03.473 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.470+0000 7fc9a3fff700 1 -- 172.21.15.131:0/1886996664 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9ac022900 con 0x7fc9bc06fcd0 2023-10-12T22:31:03.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:03 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:02.108816+0000 mgr.smithi131.efktma (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:31:03.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:03 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:31:02.109353+0000 mgr.smithi131.efktma (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-10-12T22:31:03.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:03 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:02.111552+0000 mon.smithi131 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:31:03.660 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.658+0000 7fc9c1c99700 1 -- 172.21.15.131:0/1886996664 --> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9bc10bdd0 con 0x7fc9a802fdb0 2023-10-12T22:31:03.664 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.662+0000 7fc9a3fff700 1 -- 172.21.15.131:0/1886996664 <== mgr.14114 v2:172.21.15.131:6800/3683890104 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fc9bc10bdd0 con 0x7fc9a802fdb0 2023-10-12T22:31:03.664 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled crash update... 2023-10-12T22:31:03.666 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.664+0000 7fc9c1c99700 1 -- 172.21.15.131:0/1886996664 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fc9a802fdb0 msgr2=0x7fc9a8032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:03.666 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.664+0000 7fc9c1c99700 1 --2- 172.21.15.131:0/1886996664 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fc9a802fdb0 0x7fc9a8032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc9b00096a0 tx=0x7fc9b0006b40).stop 2023-10-12T22:31:03.666 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.664+0000 7fc9c1c99700 1 -- 172.21.15.131:0/1886996664 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 msgr2=0x7fc9bc10a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:03.667 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.664+0000 7fc9c1c99700 1 --2- 172.21.15.131:0/1886996664 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 0x7fc9bc10a9f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc9ac000c00 tx=0x7fc9ac006ee0).stop 2023-10-12T22:31:03.667 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.664+0000 7fc9c1c99700 1 -- 172.21.15.131:0/1886996664 shutdown_connections 2023-10-12T22:31:03.667 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.664+0000 7fc9c1c99700 1 --2- 172.21.15.131:0/1886996664 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fc9a802fdb0 0x7fc9a8032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:03.667 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.664+0000 7fc9c1c99700 1 --2- 172.21.15.131:0/1886996664 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9bc06fcd0 0x7fc9bc10a9f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:03.668 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.664+0000 7fc9c1c99700 1 -- 172.21.15.131:0/1886996664 >> 172.21.15.131:0/1886996664 conn(0x7fc9bc0f6280 msgr2=0x7fc9bc0f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:03.668 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.664+0000 7fc9c1c99700 1 -- 172.21.15.131:0/1886996664 shutdown_connections 2023-10-12T22:31:03.668 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:03.664+0000 7fc9c1c99700 1 -- 172.21.15.131:0/1886996664 wait complete. 2023-10-12T22:31:04.122 INFO:teuthology.orchestra.run.smithi131.stderr:Enabling mgr prometheus module... 2023-10-12T22:31:04.122 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-10-12T22:31:04.484 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.482+0000 7f4017672700 1 Processor -- start 2023-10-12T22:31:04.485 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.482+0000 7f4017672700 1 -- start start 2023-10-12T22:31:04.485 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.483+0000 7f4017672700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 0x7f40100fbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:04.485 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.483+0000 7f4017672700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40100fc4c0 con 0x7f40100fbbb0 2023-10-12T22:31:04.485 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.483+0000 7f401540e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 0x7f40100fbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:04.486 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.483+0000 7f401540e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 0x7f40100fbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48176/0 (socket says 172.21.15.131:48176) 2023-10-12T22:31:04.486 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.483+0000 7f401540e700 1 -- 172.21.15.131:0/3753578479 learned_addr learned my addr 172.21.15.131:0/3753578479 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:04.486 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.483+0000 7f401540e700 1 -- 172.21.15.131:0/3753578479 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40100fc600 con 0x7f40100fbbb0 2023-10-12T22:31:04.487 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.483+0000 7f401540e700 1 --2- 172.21.15.131:0/3753578479 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 0x7f40100fbfd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f400c00b5f0 tx=0x7f400c006ee0).ready entity=mon.0 client_cookie=ba13a2a5d5edd5e server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:04.488 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.483+0000 7f3ffffff700 1 -- 172.21.15.131:0/3753578479 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f400c00d600 con 0x7f40100fbbb0 2023-10-12T22:31:04.488 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.483+0000 7f3ffffff700 1 -- 172.21.15.131:0/3753578479 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f400c00dc10 con 0x7f40100fbbb0 2023-10-12T22:31:04.489 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.483+0000 7f3ffffff700 1 -- 172.21.15.131:0/3753578479 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f400c011cd0 con 0x7f40100fbbb0 2023-10-12T22:31:04.489 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.484+0000 7f4017672700 1 -- 172.21.15.131:0/3753578479 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 msgr2=0x7f40100fbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:04.489 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.484+0000 7f4017672700 1 --2- 172.21.15.131:0/3753578479 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 0x7f40100fbfd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f400c00b5f0 tx=0x7f400c006ee0).stop 2023-10-12T22:31:04.489 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.484+0000 7f4017672700 1 -- 172.21.15.131:0/3753578479 shutdown_connections 2023-10-12T22:31:04.489 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.484+0000 7f4017672700 1 --2- 172.21.15.131:0/3753578479 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 0x7f40100fbfd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:04.490 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.484+0000 7f4017672700 1 -- 172.21.15.131:0/3753578479 >> 172.21.15.131:0/3753578479 conn(0x7f40100f73f0 msgr2=0x7f40100f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:04.490 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.484+0000 7f4017672700 1 -- 172.21.15.131:0/3753578479 shutdown_connections 2023-10-12T22:31:04.490 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.484+0000 7f4017672700 1 -- 172.21.15.131:0/3753578479 wait complete. 2023-10-12T22:31:04.490 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.485+0000 7f4017672700 1 Processor -- start 2023-10-12T22:31:04.490 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.485+0000 7f4017672700 1 -- start start 2023-10-12T22:31:04.491 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.485+0000 7f4017672700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 0x7f401010a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:04.491 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.485+0000 7f4017672700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401010aed0 con 0x7f40100fbbb0 2023-10-12T22:31:04.491 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.485+0000 7f401540e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 0x7f401010a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:04.491 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.485+0000 7f401540e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 0x7f401010a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48184/0 (socket says 172.21.15.131:48184) 2023-10-12T22:31:04.491 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.485+0000 7f401540e700 1 -- 172.21.15.131:0/3645308872 learned_addr learned my addr 172.21.15.131:0/3645308872 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:04.492 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.486+0000 7f401540e700 1 -- 172.21.15.131:0/3645308872 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c00b200 con 0x7f40100fbbb0 2023-10-12T22:31:04.492 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.486+0000 7f401540e700 1 --2- 172.21.15.131:0/3645308872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 0x7f401010a9e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f400c00dfb0 tx=0x7f400c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:04.492 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.486+0000 7f3ffdffb700 1 -- 172.21.15.131:0/3645308872 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f400c011df0 con 0x7f40100fbbb0 2023-10-12T22:31:04.492 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.486+0000 7f3ffdffb700 1 -- 172.21.15.131:0/3645308872 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f400c01b8c0 con 0x7f40100fbbb0 2023-10-12T22:31:04.492 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.486+0000 7f3ffdffb700 1 -- 172.21.15.131:0/3645308872 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f400c01a8c0 con 0x7f40100fbbb0 2023-10-12T22:31:04.493 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.486+0000 7f4017672700 1 -- 172.21.15.131:0/3645308872 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401010b0d0 con 0x7f40100fbbb0 2023-10-12T22:31:04.493 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.486+0000 7f4017672700 1 -- 172.21.15.131:0/3645308872 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401010b4e0 con 0x7f40100fbbb0 2023-10-12T22:31:04.493 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.487+0000 7f3ffdffb700 1 -- 172.21.15.131:0/3645308872 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f400c00f070 con 0x7f40100fbbb0 2023-10-12T22:31:04.493 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.487+0000 7f3ffdffb700 1 --2- 172.21.15.131:0/3645308872 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f3ff802fd50 0x7f3ff8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:04.493 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.487+0000 7f3ffdffb700 1 -- 172.21.15.131:0/3645308872 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f400c040f30 con 0x7f40100fbbb0 2023-10-12T22:31:04.494 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.487+0000 7f4014c0d700 1 --2- 172.21.15.131:0/3645308872 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f3ff802fd50 0x7f3ff8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:04.494 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.488+0000 7f4014c0d700 1 --2- 172.21.15.131:0/3645308872 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f3ff802fd50 0x7f3ff8032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f40000096a0 tx=0x7f4000006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:04.494 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.489+0000 7f4017672700 1 -- 172.21.15.131:0/3645308872 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4010059100 con 0x7f40100fbbb0 2023-10-12T22:31:04.495 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.494+0000 7f3ffdffb700 1 -- 172.21.15.131:0/3645308872 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f400c016070 con 0x7f40100fbbb0 2023-10-12T22:31:04.700 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:04.698+0000 7f4017672700 1 -- 172.21.15.131:0/3645308872 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f40100fd920 con 0x7f40100fbbb0 2023-10-12T22:31:04.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:03.660755+0000 mgr.smithi131.efktma (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:31:04.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:31:03.661391+0000 mgr.smithi131.efktma (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2023-10-12T22:31:04.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:03.663681+0000 mon.smithi131 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:31:04.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:03.930718+0000 mon.smithi131 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:31:04.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:04.154775+0000 mon.smithi131 (mon.0) 59 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:31:05.668 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:05.667+0000 7f3ffdffb700 1 -- 172.21.15.131:0/3645308872 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f400c024790 con 0x7f40100fbbb0 2023-10-12T22:31:05.668 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:05.667+0000 7f3ffdffb700 1 -- 172.21.15.131:0/3645308872 <== mon.0 v2:172.21.15.131:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7f400c005700 con 0x7f40100fbbb0 2023-10-12T22:31:05.672 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:05.670+0000 7f4017672700 1 -- 172.21.15.131:0/3645308872 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f3ff802fd50 msgr2=0x7f3ff8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:05.672 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:05.670+0000 7f4017672700 1 --2- 172.21.15.131:0/3645308872 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f3ff802fd50 0x7f3ff8032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f40000096a0 tx=0x7f4000006b40).stop 2023-10-12T22:31:05.673 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:05.670+0000 7f4017672700 1 -- 172.21.15.131:0/3645308872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 msgr2=0x7f401010a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:05.673 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:05.670+0000 7f4017672700 1 --2- 172.21.15.131:0/3645308872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 0x7f401010a9e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f400c00dfb0 tx=0x7f400c006ee0).stop 2023-10-12T22:31:05.673 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:05.670+0000 7f4017672700 1 -- 172.21.15.131:0/3645308872 shutdown_connections 2023-10-12T22:31:05.673 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:05.670+0000 7f4017672700 1 --2- 172.21.15.131:0/3645308872 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7f3ff802fd50 0x7f3ff8032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:05.673 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:05.670+0000 7f4017672700 1 --2- 172.21.15.131:0/3645308872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40100fbbb0 0x7f401010a9e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:05.674 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:05.670+0000 7f4017672700 1 -- 172.21.15.131:0/3645308872 >> 172.21.15.131:0/3645308872 conn(0x7f40100f73f0 msgr2=0x7f40100f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:05.674 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:05.671+0000 7f4017672700 1 -- 172.21.15.131:0/3645308872 shutdown_connections 2023-10-12T22:31:05.674 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:05.671+0000 7f4017672700 1 -- 172.21.15.131:0/3645308872 wait complete. 2023-10-12T22:31:05.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:05 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:04.700256+0000 mon.smithi131 (mon.0) 60 : audit [INF] from='client.? 172.21.15.131:0/3645308872' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-10-12T22:31:05.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:05 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:04.887092+0000 mon.smithi131 (mon.0) 61 : audit [INF] from='mgr.14114 172.21.15.131:0/2913702600' entity='mgr.smithi131.efktma' 2023-10-12T22:31:06.030 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying prometheus service with default placement... 2023-10-12T22:31:06.031 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-10-12T22:31:06.411 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.408+0000 7fe2c5298700 1 Processor -- start 2023-10-12T22:31:06.411 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.408+0000 7fe2c5298700 1 -- start start 2023-10-12T22:31:06.411 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.409+0000 7fe2c5298700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 0x7fe2c0100c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:06.411 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.409+0000 7fe2c5298700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00fc6a0 con 0x7fe2c006fc50 2023-10-12T22:31:06.412 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.409+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 0x7fe2c0100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:06.412 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.409+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 0x7fe2c0100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48198/0 (socket says 172.21.15.131:48198) 2023-10-12T22:31:06.412 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.409+0000 7fe2beffd700 1 -- 172.21.15.131:0/2565098185 learned_addr learned my addr 172.21.15.131:0/2565098185 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:06.414 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.409+0000 7fe2beffd700 1 -- 172.21.15.131:0/2565098185 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c0101110 con 0x7fe2c006fc50 2023-10-12T22:31:06.414 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.409+0000 7fe2beffd700 1 --2- 172.21.15.131:0/2565098185 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 0x7fe2c0100c20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fe2b000b5f0 tx=0x7fe2b0006ee0).ready entity=mon.0 client_cookie=9797f540146f0b8a server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:06.414 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.409+0000 7fe2bdffb700 1 -- 172.21.15.131:0/2565098185 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2b000d600 con 0x7fe2c006fc50 2023-10-12T22:31:06.415 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.409+0000 7fe2bdffb700 1 -- 172.21.15.131:0/2565098185 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2b000dc10 con 0x7fe2c006fc50 2023-10-12T22:31:06.415 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.409+0000 7fe2bdffb700 1 -- 172.21.15.131:0/2565098185 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2b0011cd0 con 0x7fe2c006fc50 2023-10-12T22:31:06.415 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.410+0000 7fe2c5298700 1 -- 172.21.15.131:0/2565098185 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 msgr2=0x7fe2c0100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:06.415 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.410+0000 7fe2c5298700 1 --2- 172.21.15.131:0/2565098185 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 0x7fe2c0100c20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fe2b000b5f0 tx=0x7fe2b0006ee0).stop 2023-10-12T22:31:06.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.411+0000 7fe2c5298700 1 -- 172.21.15.131:0/2565098185 shutdown_connections 2023-10-12T22:31:06.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.411+0000 7fe2c5298700 1 --2- 172.21.15.131:0/2565098185 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 0x7fe2c0100c20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:06.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.411+0000 7fe2c5298700 1 -- 172.21.15.131:0/2565098185 >> 172.21.15.131:0/2565098185 conn(0x7fe2c00f6280 msgr2=0x7fe2c00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:06.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.411+0000 7fe2c5298700 1 -- 172.21.15.131:0/2565098185 shutdown_connections 2023-10-12T22:31:06.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.411+0000 7fe2c5298700 1 -- 172.21.15.131:0/2565098185 wait complete. 2023-10-12T22:31:06.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.411+0000 7fe2c5298700 1 Processor -- start 2023-10-12T22:31:06.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.412+0000 7fe2c5298700 1 -- start start 2023-10-12T22:31:06.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.412+0000 7fe2c5298700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 0x7fe2c010cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:06.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.412+0000 7fe2c5298700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c010d220 con 0x7fe2c006fc50 2023-10-12T22:31:06.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.412+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 0x7fe2c010cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:06.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.412+0000 7fe2beffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 0x7fe2c010cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48204/0 (socket says 172.21.15.131:48204) 2023-10-12T22:31:06.418 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.412+0000 7fe2beffd700 1 -- 172.21.15.131:0/3256441336 learned_addr learned my addr 172.21.15.131:0/3256441336 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:06.418 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.412+0000 7fe2beffd700 1 -- 172.21.15.131:0/3256441336 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b000b200 con 0x7fe2c006fc50 2023-10-12T22:31:06.418 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.412+0000 7fe2beffd700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 0x7fe2c010cd30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe2b000c010 tx=0x7fe2b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:06.418 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.412+0000 7fe2a77fe700 1 -- 172.21.15.131:0/3256441336 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2b0011df0 con 0x7fe2c006fc50 2023-10-12T22:31:06.418 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.413+0000 7fe2c5298700 1 -- 172.21.15.131:0/3256441336 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c010d420 con 0x7fe2c006fc50 2023-10-12T22:31:06.418 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.413+0000 7fe2a77fe700 1 -- 172.21.15.131:0/3256441336 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2b001b8c0 con 0x7fe2c006fc50 2023-10-12T22:31:06.419 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.413+0000 7fe2a77fe700 1 -- 172.21.15.131:0/3256441336 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2b001a830 con 0x7fe2c006fc50 2023-10-12T22:31:06.419 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.413+0000 7fe2c5298700 1 -- 172.21.15.131:0/3256441336 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c010d830 con 0x7fe2c006fc50 2023-10-12T22:31:06.420 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.413+0000 7fe2a77fe700 1 -- 172.21.15.131:0/3256441336 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fe2b000f070 con 0x7fe2c006fc50 2023-10-12T22:31:06.420 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.414+0000 7fe2a77fe700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 0x7fe2ac032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:06.420 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.414+0000 7fe2c5298700 1 -- 172.21.15.131:0/3256441336 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c0045ad0 con 0x7fe2c006fc50 2023-10-12T22:31:06.420 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.414+0000 7fe2a77fe700 1 -- 172.21.15.131:0/3256441336 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe2b0040fe0 con 0x7fe2c006fc50 2023-10-12T22:31:06.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.414+0000 7fe2be7fc700 1 -- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 msgr2=0x7fe2ac032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/3683890104 2023-10-12T22:31:06.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.414+0000 7fe2be7fc700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 0x7fe2ac032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-12T22:31:06.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.418+0000 7fe2a77fe700 1 -- 172.21.15.131:0/3256441336 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2b0016030 con 0x7fe2c006fc50 2023-10-12T22:31:06.604 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.604+0000 7fe2c5298700 1 -- 172.21.15.131:0/3256441336 --> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2c010df60 con 0x7fe2ac02fdb0 2023-10-12T22:31:06.616 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.615+0000 7fe2be7fc700 1 -- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 msgr2=0x7fe2ac032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/3683890104 2023-10-12T22:31:06.616 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:06.615+0000 7fe2be7fc700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 0x7fe2ac032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-12T22:31:06.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:05.668851+0000 mon.smithi131 (mon.0) 62 : audit [INF] from='client.? 172.21.15.131:0/3645308872' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-10-12T22:31:06.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:05.668899+0000 mon.smithi131 (mon.0) 63 : cluster [DBG] mgrmap e8: smithi131.efktma(active, since 16s) 2023-10-12T22:31:07.017 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:07.016+0000 7fe2be7fc700 1 -- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 msgr2=0x7fe2ac032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/3683890104 2023-10-12T22:31:07.017 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:07.016+0000 7fe2be7fc700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 0x7fe2ac032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-12T22:31:07.819 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:07.817+0000 7fe2be7fc700 1 -- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 msgr2=0x7fe2ac032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/3683890104 2023-10-12T22:31:07.819 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:07.817+0000 7fe2be7fc700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 0x7fe2ac032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-12T22:31:09.420 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:09.419+0000 7fe2be7fc700 1 -- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 msgr2=0x7fe2ac032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/3683890104 2023-10-12T22:31:09.420 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:09.419+0000 7fe2be7fc700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 0x7fe2ac032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-10-12T22:31:10.933 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:10.932+0000 7fe2a77fe700 1 -- 172.21.15.131:0/3256441336 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fe2b0024790 con 0x7fe2c006fc50 2023-10-12T22:31:10.933 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:10.932+0000 7fe2a77fe700 1 -- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 msgr2=0x7fe2ac032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-12T22:31:10.933 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:10.932+0000 7fe2a77fe700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 0x7fe2ac032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-12T22:31:10.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:10.636005+0000 mon.smithi131 (mon.0) 64 : cluster [INF] Active manager daemon smithi131.efktma restarted 2023-10-12T22:31:10.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:10.636457+0000 mon.smithi131 (mon.0) 65 : cluster [INF] Activating manager daemon smithi131.efktma 2023-10-12T22:31:10.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:10.638089+0000 mon.smithi131 (mon.0) 66 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-10-12T22:31:11.936 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.935+0000 7fe2a77fe700 1 -- 172.21.15.131:0/3256441336 <== mon.0 v2:172.21.15.131:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fe2b0043130 con 0x7fe2c006fc50 2023-10-12T22:31:11.937 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.936+0000 7fe2a77fe700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fe2ac033c20 0x7fe2ac036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:11.937 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.936+0000 7fe2a77fe700 1 -- 172.21.15.131:0/3256441336 --> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2c010df60 con 0x7fe2ac033c20 2023-10-12T22:31:11.937 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.936+0000 7fe2be7fc700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fe2ac033c20 0x7fe2ac036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:11.938 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.937+0000 7fe2be7fc700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fe2ac033c20 0x7fe2ac036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe2b40036d0 tx=0x7fe2b4006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:11.942 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.942+0000 7fe2a77fe700 1 -- 172.21.15.131:0/3256441336 <== mgr.14142 v2:172.21.15.131:6800/3456822191 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fe2c010df60 con 0x7fe2ac033c20 2023-10-12T22:31:11.943 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-10-12T22:31:11.945 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.944+0000 7fe2c5298700 1 -- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fe2ac033c20 msgr2=0x7fe2ac036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:11.946 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.944+0000 7fe2c5298700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fe2ac033c20 0x7fe2ac036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe2b40036d0 tx=0x7fe2b4006cb0).stop 2023-10-12T22:31:11.946 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.944+0000 7fe2c5298700 1 -- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 msgr2=0x7fe2c010cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:11.946 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.944+0000 7fe2c5298700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 0x7fe2c010cd30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe2b000c010 tx=0x7fe2b0006ee0).stop 2023-10-12T22:31:11.946 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.944+0000 7fe2c5298700 1 -- 172.21.15.131:0/3256441336 shutdown_connections 2023-10-12T22:31:11.946 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.944+0000 7fe2c5298700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fe2ac033c20 0x7fe2ac036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:11.947 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.944+0000 7fe2c5298700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2c006fc50 0x7fe2c010cd30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:11.947 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.944+0000 7fe2c5298700 1 --2- 172.21.15.131:0/3256441336 >> [v2:172.21.15.131:6800/3683890104,v1:172.21.15.131:6801/3683890104] conn(0x7fe2ac02fdb0 0x7fe2ac032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-12T22:31:11.947 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.944+0000 7fe2c5298700 1 -- 172.21.15.131:0/3256441336 >> 172.21.15.131:0/3256441336 conn(0x7fe2c00f6280 msgr2=0x7fe2c00f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:11.947 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.945+0000 7fe2c5298700 1 -- 172.21.15.131:0/3256441336 shutdown_connections 2023-10-12T22:31:11.947 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:11.945+0000 7fe2c5298700 1 -- 172.21.15.131:0/3256441336 wait complete. 2023-10-12T22:31:12.103 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:10.933332+0000 mon.smithi131 (mon.0) 67 : cluster [DBG] mgrmap e9: smithi131.efktma(active, starting, since 0.296958s) 2023-10-12T22:31:12.103 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:10.934249+0000 mon.smithi131 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2023-10-12T22:31:12.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:10.934959+0000 mon.smithi131 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mgr metadata", "who": "smithi131.efktma", "id": "smithi131.efktma"}]: dispatch 2023-10-12T22:31:12.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:10.935617+0000 mon.smithi131 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-12T22:31:12.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:10.935782+0000 mon.smithi131 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-12T22:31:12.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:10.935928+0000 mon.smithi131 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-12T22:31:12.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:10.949645+0000 mon.smithi131 (mon.0) 73 : cluster [INF] Manager daemon smithi131.efktma is now available 2023-10-12T22:31:12.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:11.049056+0000 mon.smithi131 (mon.0) 74 : audit [DBG] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-12T22:31:12.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:11.050387+0000 mon.smithi131 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-12T22:31:12.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:11.065065+0000 mon.smithi131 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/mirror_snapshot_schedule"}]: dispatch 2023-10-12T22:31:12.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:11.067451+0000 mon.smithi131 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/trash_purge_schedule"}]: dispatch 2023-10-12T22:31:12.246 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying grafana service with default placement... 2023-10-12T22:31:12.246 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-10-12T22:31:12.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:11.937905+0000 mon.smithi131 (mon.0) 78 : cluster [DBG] mgrmap e10: smithi131.efktma(active, since 1.30153s) 2023-10-12T22:31:12.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:31:11.938850+0000 mgr.smithi131.efktma (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-10-12T22:31:12.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:11.942153+0000 mon.smithi131 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' 2023-10-12T22:31:12.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:11.957631+0000 mon.smithi131 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' 2023-10-12T22:31:12.999 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.996+0000 7ff3f38ab700 1 Processor -- start 2023-10-12T22:31:12.999 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.996+0000 7ff3f38ab700 1 -- start start 2023-10-12T22:31:12.999 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.996+0000 7ff3f38ab700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 0x7ff3ec0faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:12.999 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.996+0000 7ff3f38ab700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0fb480 con 0x7ff3ec0faae0 2023-10-12T22:31:13.000 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.997+0000 7ff3f1647700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 0x7ff3ec0faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:13.000 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.997+0000 7ff3f1647700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 0x7ff3ec0faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37764/0 (socket says 172.21.15.131:37764) 2023-10-12T22:31:13.000 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.997+0000 7ff3f1647700 1 -- 172.21.15.131:0/848194029 learned_addr learned my addr 172.21.15.131:0/848194029 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:13.000 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.997+0000 7ff3f1647700 1 -- 172.21.15.131:0/848194029 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec0fb5c0 con 0x7ff3ec0faae0 2023-10-12T22:31:13.000 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.997+0000 7ff3f1647700 1 --2- 172.21.15.131:0/848194029 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 0x7ff3ec0faf00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff3dc00b5f0 tx=0x7ff3dc006ee0).ready entity=mon.0 client_cookie=c77459213141039e server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:13.001 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.997+0000 7ff3dbfff700 1 -- 172.21.15.131:0/848194029 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3dc00d600 con 0x7ff3ec0faae0 2023-10-12T22:31:13.001 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.997+0000 7ff3dbfff700 1 -- 172.21.15.131:0/848194029 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3dc00dc10 con 0x7ff3ec0faae0 2023-10-12T22:31:13.001 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.997+0000 7ff3dbfff700 1 -- 172.21.15.131:0/848194029 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3dc011cd0 con 0x7ff3ec0faae0 2023-10-12T22:31:13.001 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.998+0000 7ff3f38ab700 1 -- 172.21.15.131:0/848194029 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 msgr2=0x7ff3ec0faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:13.001 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.998+0000 7ff3f38ab700 1 --2- 172.21.15.131:0/848194029 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 0x7ff3ec0faf00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff3dc00b5f0 tx=0x7ff3dc006ee0).stop 2023-10-12T22:31:13.003 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.998+0000 7ff3f38ab700 1 -- 172.21.15.131:0/848194029 shutdown_connections 2023-10-12T22:31:13.004 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.998+0000 7ff3f38ab700 1 --2- 172.21.15.131:0/848194029 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 0x7ff3ec0faf00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:13.004 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.998+0000 7ff3f38ab700 1 -- 172.21.15.131:0/848194029 >> 172.21.15.131:0/848194029 conn(0x7ff3ec0f6280 msgr2=0x7ff3ec0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:13.004 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.998+0000 7ff3f38ab700 1 -- 172.21.15.131:0/848194029 shutdown_connections 2023-10-12T22:31:13.004 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.998+0000 7ff3f38ab700 1 -- 172.21.15.131:0/848194029 wait complete. 2023-10-12T22:31:13.004 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.998+0000 7ff3f38ab700 1 Processor -- start 2023-10-12T22:31:13.004 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.998+0000 7ff3f38ab700 1 -- start start 2023-10-12T22:31:13.005 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.999+0000 7ff3f38ab700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 0x7ff3ec10d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:13.005 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.999+0000 7ff3f38ab700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec06fc70 con 0x7ff3ec0faae0 2023-10-12T22:31:13.005 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.999+0000 7ff3f1647700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 0x7ff3ec10d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:13.005 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.999+0000 7ff3f1647700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 0x7ff3ec10d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37778/0 (socket says 172.21.15.131:37778) 2023-10-12T22:31:13.005 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.999+0000 7ff3f1647700 1 -- 172.21.15.131:0/2169807044 learned_addr learned my addr 172.21.15.131:0/2169807044 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:13.006 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.999+0000 7ff3f1647700 1 -- 172.21.15.131:0/2169807044 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc00b200 con 0x7ff3ec0faae0 2023-10-12T22:31:13.006 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.999+0000 7ff3f1647700 1 --2- 172.21.15.131:0/2169807044 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 0x7ff3ec10d750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ff3dc006800 tx=0x7ff3dc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:13.006 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.999+0000 7ff3d9ffb700 1 -- 172.21.15.131:0/2169807044 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3dc011df0 con 0x7ff3ec0faae0 2023-10-12T22:31:13.006 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.999+0000 7ff3d9ffb700 1 -- 172.21.15.131:0/2169807044 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3dc01b8c0 con 0x7ff3ec0faae0 2023-10-12T22:31:13.006 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.999+0000 7ff3d9ffb700 1 -- 172.21.15.131:0/2169807044 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3dc01a800 con 0x7ff3ec0faae0 2023-10-12T22:31:13.007 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.999+0000 7ff3f38ab700 1 -- 172.21.15.131:0/2169807044 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec06fe70 con 0x7ff3ec0faae0 2023-10-12T22:31:13.007 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:12.999+0000 7ff3f38ab700 1 -- 172.21.15.131:0/2169807044 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec10de00 con 0x7ff3ec0faae0 2023-10-12T22:31:13.007 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.000+0000 7ff3d9ffb700 1 -- 172.21.15.131:0/2169807044 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ff3dc00f070 con 0x7ff3ec0faae0 2023-10-12T22:31:13.007 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.000+0000 7ff3d9ffb700 1 --2- 172.21.15.131:0/2169807044 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7ff3d402feb0 0x7ff3d4032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:13.007 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.000+0000 7ff3d9ffb700 1 -- 172.21.15.131:0/2169807044 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff3dc040ff0 con 0x7ff3ec0faae0 2023-10-12T22:31:13.007 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.000+0000 7ff3f0e46700 1 --2- 172.21.15.131:0/2169807044 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7ff3d402feb0 0x7ff3d4032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:13.008 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.001+0000 7ff3f38ab700 1 -- 172.21.15.131:0/2169807044 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec045ad0 con 0x7ff3ec0faae0 2023-10-12T22:31:13.008 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.001+0000 7ff3f0e46700 1 --2- 172.21.15.131:0/2169807044 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7ff3d402feb0 0x7ff3d4032370 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff3e80096a0 tx=0x7ff3e8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:13.008 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.005+0000 7ff3d9ffb700 1 -- 172.21.15.131:0/2169807044 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3dc016080 con 0x7ff3ec0faae0 2023-10-12T22:31:13.186 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.184+0000 7ff3f38ab700 1 -- 172.21.15.131:0/2169807044 --> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3ec0008d0 con 0x7ff3d402feb0 2023-10-12T22:31:13.189 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.188+0000 7ff3d9ffb700 1 -- 172.21.15.131:0/2169807044 <== mgr.14142 v2:172.21.15.131:6800/3456822191 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7ff3ec0008d0 con 0x7ff3d402feb0 2023-10-12T22:31:13.190 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-10-12T22:31:13.192 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.190+0000 7ff3f38ab700 1 -- 172.21.15.131:0/2169807044 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7ff3d402feb0 msgr2=0x7ff3d4032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:13.192 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.190+0000 7ff3f38ab700 1 --2- 172.21.15.131:0/2169807044 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7ff3d402feb0 0x7ff3d4032370 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff3e80096a0 tx=0x7ff3e8006b40).stop 2023-10-12T22:31:13.192 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.190+0000 7ff3f38ab700 1 -- 172.21.15.131:0/2169807044 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 msgr2=0x7ff3ec10d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:13.193 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.190+0000 7ff3f38ab700 1 --2- 172.21.15.131:0/2169807044 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 0x7ff3ec10d750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ff3dc006800 tx=0x7ff3dc006ee0).stop 2023-10-12T22:31:13.193 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.190+0000 7ff3f38ab700 1 -- 172.21.15.131:0/2169807044 shutdown_connections 2023-10-12T22:31:13.193 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.190+0000 7ff3f38ab700 1 --2- 172.21.15.131:0/2169807044 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7ff3d402feb0 0x7ff3d4032370 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:13.193 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.190+0000 7ff3f38ab700 1 --2- 172.21.15.131:0/2169807044 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3ec0faae0 0x7ff3ec10d750 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:13.193 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.190+0000 7ff3f38ab700 1 -- 172.21.15.131:0/2169807044 >> 172.21.15.131:0/2169807044 conn(0x7ff3ec0f6280 msgr2=0x7ff3ec0f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:13.194 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.191+0000 7ff3f38ab700 1 -- 172.21.15.131:0/2169807044 shutdown_connections 2023-10-12T22:31:13.194 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.191+0000 7ff3f38ab700 1 -- 172.21.15.131:0/2169807044 wait complete. 2023-10-12T22:31:13.562 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying node-exporter service with default placement... 2023-10-12T22:31:13.563 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-10-12T22:31:13.891 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.888+0000 7fd203fff700 1 Processor -- start 2023-10-12T22:31:13.891 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.888+0000 7fd203fff700 1 -- start start 2023-10-12T22:31:13.892 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.888+0000 7fd203fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd20406fb50 0x7fd2040fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:13.892 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.888+0000 7fd203fff700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20406ff70 con 0x7fd20406fb50 2023-10-12T22:31:13.892 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.888+0000 7fd202ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd20406fb50 0x7fd2040fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:13.892 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.888+0000 7fd202ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd20406fb50 0x7fd2040fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37790/0 (socket says 172.21.15.131:37790) 2023-10-12T22:31:13.892 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.888+0000 7fd202ffd700 1 -- 172.21.15.131:0/1912810791 learned_addr learned my addr 172.21.15.131:0/1912810791 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:13.893 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.888+0000 7fd202ffd700 1 -- 172.21.15.131:0/1912810791 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2040fec20 con 0x7fd20406fb50 2023-10-12T22:31:13.893 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.888+0000 7fd202ffd700 1 --2- 172.21.15.131:0/1912810791 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd20406fb50 0x7fd2040fe730 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fd1f400b5f0 tx=0x7fd1f4006ee0).ready entity=mon.0 client_cookie=c3bc0b1cce0e2500 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:13.893 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.889+0000 7fd201ffb700 1 -- 172.21.15.131:0/1912810791 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1f400d600 con 0x7fd20406fb50 2023-10-12T22:31:13.893 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.889+0000 7fd201ffb700 1 -- 172.21.15.131:0/1912810791 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1f400dc10 con 0x7fd20406fb50 2023-10-12T22:31:13.893 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.889+0000 7fd201ffb700 1 -- 172.21.15.131:0/1912810791 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1f4011cd0 con 0x7fd20406fb50 2023-10-12T22:31:13.894 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.889+0000 7fd203fff700 1 -- 172.21.15.131:0/1912810791 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd20406fb50 msgr2=0x7fd2040fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:13.894 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.889+0000 7fd203fff700 1 --2- 172.21.15.131:0/1912810791 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd20406fb50 0x7fd2040fe730 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fd1f400b5f0 tx=0x7fd1f4006ee0).stop 2023-10-12T22:31:13.894 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.889+0000 7fd203fff700 1 -- 172.21.15.131:0/1912810791 shutdown_connections 2023-10-12T22:31:13.894 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.889+0000 7fd203fff700 1 --2- 172.21.15.131:0/1912810791 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd20406fb50 0x7fd2040fe730 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:13.894 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.889+0000 7fd203fff700 1 -- 172.21.15.131:0/1912810791 >> 172.21.15.131:0/1912810791 conn(0x7fd2040f6130 msgr2=0x7fd2040f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:13.895 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.889+0000 7fd203fff700 1 -- 172.21.15.131:0/1912810791 shutdown_connections 2023-10-12T22:31:13.895 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.889+0000 7fd203fff700 1 -- 172.21.15.131:0/1912810791 wait complete. 2023-10-12T22:31:13.895 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.890+0000 7fd203fff700 1 Processor -- start 2023-10-12T22:31:13.895 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.890+0000 7fd203fff700 1 -- start start 2023-10-12T22:31:13.895 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.890+0000 7fd203fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2040fc170 0x7fd2040fa630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:13.896 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.890+0000 7fd203fff700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2040fc590 con 0x7fd2040fc170 2023-10-12T22:31:13.898 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.890+0000 7fd202ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2040fc170 0x7fd2040fa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:13.898 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.890+0000 7fd202ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2040fc170 0x7fd2040fa630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37800/0 (socket says 172.21.15.131:37800) 2023-10-12T22:31:13.898 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.890+0000 7fd202ffd700 1 -- 172.21.15.131:0/2908798556 learned_addr learned my addr 172.21.15.131:0/2908798556 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:13.898 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.891+0000 7fd202ffd700 1 -- 172.21.15.131:0/2908798556 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f400b200 con 0x7fd2040fc170 2023-10-12T22:31:13.899 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.891+0000 7fd202ffd700 1 --2- 172.21.15.131:0/2908798556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2040fc170 0x7fd2040fa630 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fd20406f920 tx=0x7fd1f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:13.899 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.891+0000 7fd1eb7fe700 1 -- 172.21.15.131:0/2908798556 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1f4011d80 con 0x7fd2040fc170 2023-10-12T22:31:13.899 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.891+0000 7fd203fff700 1 -- 172.21.15.131:0/2908798556 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2040fab20 con 0x7fd2040fc170 2023-10-12T22:31:13.899 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.891+0000 7fd1eb7fe700 1 -- 172.21.15.131:0/2908798556 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1f401b8c0 con 0x7fd2040fc170 2023-10-12T22:31:13.899 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.891+0000 7fd1eb7fe700 1 -- 172.21.15.131:0/2908798556 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1f401a7b0 con 0x7fd2040fc170 2023-10-12T22:31:13.899 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.891+0000 7fd203fff700 1 -- 172.21.15.131:0/2908798556 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2040faf90 con 0x7fd2040fc170 2023-10-12T22:31:13.900 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.892+0000 7fd1eb7fe700 1 -- 172.21.15.131:0/2908798556 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fd1f400f070 con 0x7fd2040fc170 2023-10-12T22:31:13.900 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.892+0000 7fd203fff700 1 -- 172.21.15.131:0/2908798556 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd204045ad0 con 0x7fd2040fc170 2023-10-12T22:31:13.900 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.892+0000 7fd1eb7fe700 1 --2- 172.21.15.131:0/2908798556 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fd1f002fe60 0x7fd1f0032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:13.900 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.892+0000 7fd1eb7fe700 1 -- 172.21.15.131:0/2908798556 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd1f4041150 con 0x7fd2040fc170 2023-10-12T22:31:13.900 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.892+0000 7fd2027fc700 1 --2- 172.21.15.131:0/2908798556 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fd1f002fe60 0x7fd1f0032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:13.901 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.892+0000 7fd2027fc700 1 --2- 172.21.15.131:0/2908798556 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fd1f002fe60 0x7fd1f0032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd1f80096a0 tx=0x7fd1f8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:13.901 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:13.896+0000 7fd1eb7fe700 1 -- 172.21.15.131:0/2908798556 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1f4010360 con 0x7fd2040fc170 2023-10-12T22:31:14.076 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.074+0000 7fd203fff700 1 -- 172.21.15.131:0/2908798556 --> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fd204057f10 con 0x7fd1f002fe60 2023-10-12T22:31:14.079 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.078+0000 7fd1eb7fe700 1 -- 172.21.15.131:0/2908798556 <== mgr.14142 v2:172.21.15.131:6800/3456822191 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fd204057f10 con 0x7fd1f002fe60 2023-10-12T22:31:14.080 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-10-12T22:31:14.082 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.080+0000 7fd203fff700 1 -- 172.21.15.131:0/2908798556 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fd1f002fe60 msgr2=0x7fd1f0032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:14.082 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.080+0000 7fd203fff700 1 --2- 172.21.15.131:0/2908798556 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fd1f002fe60 0x7fd1f0032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd1f80096a0 tx=0x7fd1f8006b40).stop 2023-10-12T22:31:14.083 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.080+0000 7fd203fff700 1 -- 172.21.15.131:0/2908798556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2040fc170 msgr2=0x7fd2040fa630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:14.083 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.080+0000 7fd203fff700 1 --2- 172.21.15.131:0/2908798556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2040fc170 0x7fd2040fa630 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fd20406f920 tx=0x7fd1f4006ee0).stop 2023-10-12T22:31:14.083 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.080+0000 7fd203fff700 1 -- 172.21.15.131:0/2908798556 shutdown_connections 2023-10-12T22:31:14.083 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.080+0000 7fd203fff700 1 --2- 172.21.15.131:0/2908798556 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fd1f002fe60 0x7fd1f0032320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:14.083 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.080+0000 7fd203fff700 1 --2- 172.21.15.131:0/2908798556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2040fc170 0x7fd2040fa630 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:14.084 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.080+0000 7fd203fff700 1 -- 172.21.15.131:0/2908798556 >> 172.21.15.131:0/2908798556 conn(0x7fd2040f6130 msgr2=0x7fd2040f7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:14.084 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.080+0000 7fd203fff700 1 -- 172.21.15.131:0/2908798556 shutdown_connections 2023-10-12T22:31:14.084 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.080+0000 7fd203fff700 1 -- 172.21.15.131:0/2908798556 wait complete. 2023-10-12T22:31:14.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:13 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:12.959877+0000 mon.smithi131 (mon.0) 81 : cluster [DBG] mgrmap e11: smithi131.efktma(active, since 2s) 2023-10-12T22:31:14.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:13 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:13.186635+0000 mgr.smithi131.efktma (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:31:14.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:13 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:31:13.187120+0000 mgr.smithi131.efktma (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-10-12T22:31:14.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:13 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:13.189423+0000 mon.smithi131 (mon.0) 82 : audit [INF] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' 2023-10-12T22:31:14.466 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying alertmanager service with default placement... 2023-10-12T22:31:14.467 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-10-12T22:31:14.793 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.790+0000 7fcde2f5d700 1 Processor -- start 2023-10-12T22:31:14.793 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.790+0000 7fcde2f5d700 1 -- start start 2023-10-12T22:31:14.793 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.790+0000 7fcde2f5d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 0x7fcddc0fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:14.793 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.790+0000 7fcde2f5d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0fed60 con 0x7fcddc06fc90 2023-10-12T22:31:14.794 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.790+0000 7fcde0cf9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 0x7fcddc0fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:14.794 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.791+0000 7fcde0cf9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 0x7fcddc0fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37808/0 (socket says 172.21.15.131:37808) 2023-10-12T22:31:14.794 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.791+0000 7fcde0cf9700 1 -- 172.21.15.131:0/738987890 learned_addr learned my addr 172.21.15.131:0/738987890 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:14.794 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.791+0000 7fcde0cf9700 1 -- 172.21.15.131:0/738987890 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc0feea0 con 0x7fcddc06fc90 2023-10-12T22:31:14.794 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.791+0000 7fcde0cf9700 1 --2- 172.21.15.131:0/738987890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 0x7fcddc0fe7e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fcdd000b5f0 tx=0x7fcdd0006ee0).ready entity=mon.0 client_cookie=631cecb9f3810807 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:14.796 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.791+0000 7fcddb7fe700 1 -- 172.21.15.131:0/738987890 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdd000d600 con 0x7fcddc06fc90 2023-10-12T22:31:14.796 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.791+0000 7fcddb7fe700 1 -- 172.21.15.131:0/738987890 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdd000dc10 con 0x7fcddc06fc90 2023-10-12T22:31:14.796 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.791+0000 7fcddb7fe700 1 -- 172.21.15.131:0/738987890 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdd0011cd0 con 0x7fcddc06fc90 2023-10-12T22:31:14.796 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.792+0000 7fcde2f5d700 1 -- 172.21.15.131:0/738987890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 msgr2=0x7fcddc0fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:14.796 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.792+0000 7fcde2f5d700 1 --2- 172.21.15.131:0/738987890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 0x7fcddc0fe7e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fcdd000b5f0 tx=0x7fcdd0006ee0).stop 2023-10-12T22:31:14.797 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.792+0000 7fcde2f5d700 1 -- 172.21.15.131:0/738987890 shutdown_connections 2023-10-12T22:31:14.797 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.792+0000 7fcde2f5d700 1 --2- 172.21.15.131:0/738987890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 0x7fcddc0fe7e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:14.797 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.792+0000 7fcde2f5d700 1 -- 172.21.15.131:0/738987890 >> 172.21.15.131:0/738987890 conn(0x7fcddc0f6280 msgr2=0x7fcddc0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:14.797 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.792+0000 7fcde2f5d700 1 -- 172.21.15.131:0/738987890 shutdown_connections 2023-10-12T22:31:14.797 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.792+0000 7fcde2f5d700 1 -- 172.21.15.131:0/738987890 wait complete. 2023-10-12T22:31:14.798 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.792+0000 7fcde2f5d700 1 Processor -- start 2023-10-12T22:31:14.798 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.792+0000 7fcde2f5d700 1 -- start start 2023-10-12T22:31:14.798 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.792+0000 7fcde2f5d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 0x7fcddc10a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:14.798 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.793+0000 7fcde2f5d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc10aed0 con 0x7fcddc06fc90 2023-10-12T22:31:14.798 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.793+0000 7fcde0cf9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 0x7fcddc10a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:14.799 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.793+0000 7fcde0cf9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 0x7fcddc10a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37818/0 (socket says 172.21.15.131:37818) 2023-10-12T22:31:14.799 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.793+0000 7fcde0cf9700 1 -- 172.21.15.131:0/2026475450 learned_addr learned my addr 172.21.15.131:0/2026475450 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:14.799 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.793+0000 7fcde0cf9700 1 -- 172.21.15.131:0/2026475450 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd000b200 con 0x7fcddc06fc90 2023-10-12T22:31:14.799 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.793+0000 7fcde0cf9700 1 --2- 172.21.15.131:0/2026475450 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 0x7fcddc10a9e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fcdd0006cb0 tx=0x7fcdd0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:14.800 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.793+0000 7fcdd97fa700 1 -- 172.21.15.131:0/2026475450 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdd0011df0 con 0x7fcddc06fc90 2023-10-12T22:31:14.800 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.793+0000 7fcdd97fa700 1 -- 172.21.15.131:0/2026475450 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdd001b8c0 con 0x7fcddc06fc90 2023-10-12T22:31:14.800 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.793+0000 7fcde2f5d700 1 -- 172.21.15.131:0/2026475450 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcddc10b0d0 con 0x7fcddc06fc90 2023-10-12T22:31:14.800 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.793+0000 7fcdd97fa700 1 -- 172.21.15.131:0/2026475450 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdd001a930 con 0x7fcddc06fc90 2023-10-12T22:31:14.800 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.793+0000 7fcde2f5d700 1 -- 172.21.15.131:0/2026475450 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcddc10b4e0 con 0x7fcddc06fc90 2023-10-12T22:31:14.800 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.794+0000 7fcdd97fa700 1 -- 172.21.15.131:0/2026475450 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fcdd000f070 con 0x7fcddc06fc90 2023-10-12T22:31:14.801 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.794+0000 7fcde2f5d700 1 -- 172.21.15.131:0/2026475450 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcddc045ad0 con 0x7fcddc06fc90 2023-10-12T22:31:14.801 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.794+0000 7fcdd97fa700 1 --2- 172.21.15.131:0/2026475450 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fcdc002feb0 0x7fcdc0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:14.801 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.794+0000 7fcdd97fa700 1 -- 172.21.15.131:0/2026475450 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fcdd0040f80 con 0x7fcddc06fc90 2023-10-12T22:31:14.801 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.794+0000 7fcddbfff700 1 --2- 172.21.15.131:0/2026475450 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fcdc002feb0 0x7fcdc0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:14.801 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.795+0000 7fcddbfff700 1 --2- 172.21.15.131:0/2026475450 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fcdc002feb0 0x7fcdc0032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fcdc80096a0 tx=0x7fcdc8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:14.802 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.798+0000 7fcdd97fa700 1 -- 172.21.15.131:0/2026475450 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcdd0010be0 con 0x7fcddc06fc90 2023-10-12T22:31:14.978 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.976+0000 7fcde2f5d700 1 -- 172.21.15.131:0/2026475450 --> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fcddc10bdc0 con 0x7fcdc002feb0 2023-10-12T22:31:14.981 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.980+0000 7fcdd97fa700 1 -- 172.21.15.131:0/2026475450 <== mgr.14142 v2:172.21.15.131:6800/3456822191 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fcddc10bdc0 con 0x7fcdc002feb0 2023-10-12T22:31:14.981 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-10-12T22:31:14.984 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.982+0000 7fcde2f5d700 1 -- 172.21.15.131:0/2026475450 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fcdc002feb0 msgr2=0x7fcdc0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:14.984 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.982+0000 7fcde2f5d700 1 --2- 172.21.15.131:0/2026475450 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fcdc002feb0 0x7fcdc0032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fcdc80096a0 tx=0x7fcdc8006b40).stop 2023-10-12T22:31:14.984 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.982+0000 7fcde2f5d700 1 -- 172.21.15.131:0/2026475450 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 msgr2=0x7fcddc10a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:14.984 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.982+0000 7fcde2f5d700 1 --2- 172.21.15.131:0/2026475450 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 0x7fcddc10a9e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fcdd0006cb0 tx=0x7fcdd0006ee0).stop 2023-10-12T22:31:14.984 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.982+0000 7fcde2f5d700 1 -- 172.21.15.131:0/2026475450 shutdown_connections 2023-10-12T22:31:14.985 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.982+0000 7fcde2f5d700 1 --2- 172.21.15.131:0/2026475450 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7fcdc002feb0 0x7fcdc0032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:14.985 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.982+0000 7fcde2f5d700 1 --2- 172.21.15.131:0/2026475450 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcddc06fc90 0x7fcddc10a9e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:14.985 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.982+0000 7fcde2f5d700 1 -- 172.21.15.131:0/2026475450 >> 172.21.15.131:0/2026475450 conn(0x7fcddc0f6280 msgr2=0x7fcddc0f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:14.985 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.982+0000 7fcde2f5d700 1 -- 172.21.15.131:0/2026475450 shutdown_connections 2023-10-12T22:31:14.985 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:14.982+0000 7fcde2f5d700 1 -- 172.21.15.131:0/2026475450 wait complete. 2023-10-12T22:31:15.272 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:15 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:14.076476+0000 mgr.smithi131.efktma (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:31:15.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:15 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:31:14.077140+0000 mgr.smithi131.efktma (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-10-12T22:31:15.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:15 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:14.079470+0000 mon.smithi131 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' 2023-10-12T22:31:15.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:15 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:14.981295+0000 mon.smithi131 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.131:0/4017196291' entity='mgr.smithi131.efktma' 2023-10-12T22:31:15.328 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-10-12T22:31:15.662 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.659+0000 7f4b6e34b700 1 Processor -- start 2023-10-12T22:31:15.662 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.659+0000 7f4b6e34b700 1 -- start start 2023-10-12T22:31:15.663 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.659+0000 7f4b6e34b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 0x7f4b680fe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:15.663 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.659+0000 7f4b6e34b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b680fed80 con 0x7f4b6806fc70 2023-10-12T22:31:15.663 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.660+0000 7f4b67fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 0x7f4b680fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:15.664 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.660+0000 7f4b67fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 0x7f4b680fe800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37820/0 (socket says 172.21.15.131:37820) 2023-10-12T22:31:15.664 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.660+0000 7f4b67fff700 1 -- 172.21.15.131:0/851069125 learned_addr learned my addr 172.21.15.131:0/851069125 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:15.664 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.660+0000 7f4b67fff700 1 -- 172.21.15.131:0/851069125 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b680feec0 con 0x7f4b6806fc70 2023-10-12T22:31:15.664 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.660+0000 7f4b67fff700 1 --2- 172.21.15.131:0/851069125 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 0x7f4b680fe800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f4b5800b5f0 tx=0x7f4b58006ee0).ready entity=mon.0 client_cookie=a6485f7619c7b1dd server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:15.665 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.660+0000 7f4b66ffd700 1 -- 172.21.15.131:0/851069125 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b5800d600 con 0x7f4b6806fc70 2023-10-12T22:31:15.665 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.660+0000 7f4b66ffd700 1 -- 172.21.15.131:0/851069125 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b5800dc10 con 0x7f4b6806fc70 2023-10-12T22:31:15.665 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.660+0000 7f4b66ffd700 1 -- 172.21.15.131:0/851069125 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b58011cd0 con 0x7f4b6806fc70 2023-10-12T22:31:15.669 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.661+0000 7f4b6e34b700 1 -- 172.21.15.131:0/851069125 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 msgr2=0x7f4b680fe800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:15.670 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.661+0000 7f4b6e34b700 1 --2- 172.21.15.131:0/851069125 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 0x7f4b680fe800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f4b5800b5f0 tx=0x7f4b58006ee0).stop 2023-10-12T22:31:15.670 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.661+0000 7f4b6e34b700 1 -- 172.21.15.131:0/851069125 shutdown_connections 2023-10-12T22:31:15.670 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.661+0000 7f4b6e34b700 1 --2- 172.21.15.131:0/851069125 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 0x7f4b680fe800 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:15.671 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.661+0000 7f4b6e34b700 1 -- 172.21.15.131:0/851069125 >> 172.21.15.131:0/851069125 conn(0x7f4b680f6280 msgr2=0x7f4b680f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:15.671 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.661+0000 7f4b6e34b700 1 -- 172.21.15.131:0/851069125 shutdown_connections 2023-10-12T22:31:15.671 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.661+0000 7f4b6e34b700 1 -- 172.21.15.131:0/851069125 wait complete. 2023-10-12T22:31:15.672 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.662+0000 7f4b6e34b700 1 Processor -- start 2023-10-12T22:31:15.672 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.662+0000 7f4b6e34b700 1 -- start start 2023-10-12T22:31:15.672 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.662+0000 7f4b6e34b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 0x7f4b6810aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:15.673 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.662+0000 7f4b6e34b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6810aef0 con 0x7f4b6806fc70 2023-10-12T22:31:15.673 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.662+0000 7f4b67fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 0x7f4b6810aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:15.673 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.662+0000 7f4b67fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 0x7f4b6810aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37824/0 (socket says 172.21.15.131:37824) 2023-10-12T22:31:15.674 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.662+0000 7f4b67fff700 1 -- 172.21.15.131:0/298616286 learned_addr learned my addr 172.21.15.131:0/298616286 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:15.674 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.663+0000 7f4b67fff700 1 -- 172.21.15.131:0/298616286 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5800b200 con 0x7f4b6806fc70 2023-10-12T22:31:15.675 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.663+0000 7f4b67fff700 1 --2- 172.21.15.131:0/298616286 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 0x7f4b6810aa00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f4b58006cb0 tx=0x7f4b58006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:15.675 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.663+0000 7f4b64ff9700 1 -- 172.21.15.131:0/298616286 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b58011df0 con 0x7f4b6806fc70 2023-10-12T22:31:15.675 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.663+0000 7f4b64ff9700 1 -- 172.21.15.131:0/298616286 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b5801b8c0 con 0x7f4b6806fc70 2023-10-12T22:31:15.676 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.663+0000 7f4b64ff9700 1 -- 172.21.15.131:0/298616286 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b5801a930 con 0x7f4b6806fc70 2023-10-12T22:31:15.676 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.663+0000 7f4b6e34b700 1 -- 172.21.15.131:0/298616286 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6810b0f0 con 0x7f4b6806fc70 2023-10-12T22:31:15.676 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.663+0000 7f4b6e34b700 1 -- 172.21.15.131:0/298616286 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6810b500 con 0x7f4b6806fc70 2023-10-12T22:31:15.677 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.664+0000 7f4b64ff9700 1 -- 172.21.15.131:0/298616286 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f4b5800f070 con 0x7f4b6806fc70 2023-10-12T22:31:15.677 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.664+0000 7f4b64ff9700 1 --2- 172.21.15.131:0/298616286 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f4b5402feb0 0x7f4b54032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:15.677 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.664+0000 7f4b64ff9700 1 -- 172.21.15.131:0/298616286 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4b58040f10 con 0x7f4b6806fc70 2023-10-12T22:31:15.677 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.664+0000 7f4b6e34b700 1 -- 172.21.15.131:0/298616286 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b68045ad0 con 0x7f4b6806fc70 2023-10-12T22:31:15.677 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.664+0000 7f4b677fe700 1 --2- 172.21.15.131:0/298616286 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f4b5402feb0 0x7f4b54032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:15.678 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.664+0000 7f4b677fe700 1 --2- 172.21.15.131:0/298616286 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f4b5402feb0 0x7f4b54032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4b5c0096a0 tx=0x7f4b5c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:15.678 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.668+0000 7f4b64ff9700 1 -- 172.21.15.131:0/298616286 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b58024410 con 0x7f4b6806fc70 2023-10-12T22:31:15.821 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.820+0000 7f4b6e34b700 1 -- 172.21.15.131:0/298616286 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f4b680ffc60 con 0x7f4b6806fc70 2023-10-12T22:31:15.824 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.823+0000 7f4b64ff9700 1 -- 172.21.15.131:0/298616286 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f4b580107d0 con 0x7f4b6806fc70 2023-10-12T22:31:15.827 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.825+0000 7f4b6e34b700 1 -- 172.21.15.131:0/298616286 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f4b5402feb0 msgr2=0x7f4b54032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:15.827 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.825+0000 7f4b6e34b700 1 --2- 172.21.15.131:0/298616286 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f4b5402feb0 0x7f4b54032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4b5c0096a0 tx=0x7f4b5c006b40).stop 2023-10-12T22:31:15.827 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.825+0000 7f4b6e34b700 1 -- 172.21.15.131:0/298616286 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 msgr2=0x7f4b6810aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:15.827 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.825+0000 7f4b6e34b700 1 --2- 172.21.15.131:0/298616286 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 0x7f4b6810aa00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f4b58006cb0 tx=0x7f4b58006ee0).stop 2023-10-12T22:31:15.828 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.825+0000 7f4b6e34b700 1 -- 172.21.15.131:0/298616286 shutdown_connections 2023-10-12T22:31:15.828 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.825+0000 7f4b6e34b700 1 --2- 172.21.15.131:0/298616286 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f4b5402feb0 0x7f4b54032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:15.828 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.825+0000 7f4b6e34b700 1 --2- 172.21.15.131:0/298616286 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b6806fc70 0x7f4b6810aa00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:15.828 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.825+0000 7f4b6e34b700 1 -- 172.21.15.131:0/298616286 >> 172.21.15.131:0/298616286 conn(0x7f4b680f6280 msgr2=0x7f4b680f7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:15.828 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.826+0000 7f4b6e34b700 1 -- 172.21.15.131:0/298616286 shutdown_connections 2023-10-12T22:31:15.829 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:15.826+0000 7f4b6e34b700 1 -- 172.21.15.131:0/298616286 wait complete. 2023-10-12T22:31:16.138 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:16 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:14.978720+0000 mgr.smithi131.efktma (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:31:16.138 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:16 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:31:14.979270+0000 mgr.smithi131.efktma (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-10-12T22:31:16.138 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:16 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:15.824195+0000 mon.smithi131 (mon.0) 85 : audit [INF] from='client.? 172.21.15.131:0/298616286' entity='client.admin' 2023-10-12T22:31:16.195 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-10-12T22:31:16.538 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.534+0000 7f98b6441700 1 Processor -- start 2023-10-12T22:31:16.538 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.534+0000 7f98b6441700 1 -- start start 2023-10-12T22:31:16.538 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.534+0000 7f98b6441700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 0x7f98b00faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:16.539 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.534+0000 7f98b6441700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b00fb480 con 0x7f98b00faae0 2023-10-12T22:31:16.539 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.535+0000 7f98affff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 0x7f98b00faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:16.539 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.535+0000 7f98affff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 0x7f98b00faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37826/0 (socket says 172.21.15.131:37826) 2023-10-12T22:31:16.539 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.535+0000 7f98affff700 1 -- 172.21.15.131:0/1346175818 learned_addr learned my addr 172.21.15.131:0/1346175818 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:16.539 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.535+0000 7f98affff700 1 -- 172.21.15.131:0/1346175818 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b00fb5c0 con 0x7f98b00faae0 2023-10-12T22:31:16.540 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.535+0000 7f98affff700 1 --2- 172.21.15.131:0/1346175818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 0x7f98b00faf00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f98a000b5f0 tx=0x7f98a0006ee0).ready entity=mon.0 client_cookie=f05042d6096a9c7f server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:16.540 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.535+0000 7f98aeffd700 1 -- 172.21.15.131:0/1346175818 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98a000d600 con 0x7f98b00faae0 2023-10-12T22:31:16.540 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.535+0000 7f98aeffd700 1 -- 172.21.15.131:0/1346175818 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98a000dc10 con 0x7f98b00faae0 2023-10-12T22:31:16.540 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.535+0000 7f98aeffd700 1 -- 172.21.15.131:0/1346175818 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98a0011cd0 con 0x7f98b00faae0 2023-10-12T22:31:16.540 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.536+0000 7f98b6441700 1 -- 172.21.15.131:0/1346175818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 msgr2=0x7f98b00faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:16.541 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.536+0000 7f98b6441700 1 --2- 172.21.15.131:0/1346175818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 0x7f98b00faf00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f98a000b5f0 tx=0x7f98a0006ee0).stop 2023-10-12T22:31:16.541 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.536+0000 7f98b6441700 1 -- 172.21.15.131:0/1346175818 shutdown_connections 2023-10-12T22:31:16.541 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.536+0000 7f98b6441700 1 --2- 172.21.15.131:0/1346175818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 0x7f98b00faf00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:16.541 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.536+0000 7f98b6441700 1 -- 172.21.15.131:0/1346175818 >> 172.21.15.131:0/1346175818 conn(0x7f98b00f6280 msgr2=0x7f98b00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:16.542 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.536+0000 7f98b6441700 1 -- 172.21.15.131:0/1346175818 shutdown_connections 2023-10-12T22:31:16.542 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.536+0000 7f98b6441700 1 -- 172.21.15.131:0/1346175818 wait complete. 2023-10-12T22:31:16.542 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.537+0000 7f98b6441700 1 Processor -- start 2023-10-12T22:31:16.543 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.537+0000 7f98b6441700 1 -- start start 2023-10-12T22:31:16.543 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.537+0000 7f98b6441700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 0x7f98b010aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:16.544 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.537+0000 7f98b6441700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b010af20 con 0x7f98b00faae0 2023-10-12T22:31:16.547 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.537+0000 7f98affff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 0x7f98b010aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:16.548 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.537+0000 7f98affff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 0x7f98b010aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37838/0 (socket says 172.21.15.131:37838) 2023-10-12T22:31:16.548 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.537+0000 7f98affff700 1 -- 172.21.15.131:0/1960707982 learned_addr learned my addr 172.21.15.131:0/1960707982 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:16.548 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.537+0000 7f98affff700 1 -- 172.21.15.131:0/1960707982 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a000b200 con 0x7f98b00faae0 2023-10-12T22:31:16.548 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.537+0000 7f98affff700 1 --2- 172.21.15.131:0/1960707982 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 0x7f98b010aa30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f98a0006800 tx=0x7f98a0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:16.549 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.538+0000 7f98acff9700 1 -- 172.21.15.131:0/1960707982 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98a0011df0 con 0x7f98b00faae0 2023-10-12T22:31:16.549 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.538+0000 7f98b6441700 1 -- 172.21.15.131:0/1960707982 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b010b120 con 0x7f98b00faae0 2023-10-12T22:31:16.549 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.538+0000 7f98acff9700 1 -- 172.21.15.131:0/1960707982 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98a001b8c0 con 0x7f98b00faae0 2023-10-12T22:31:16.550 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.538+0000 7f98acff9700 1 -- 172.21.15.131:0/1960707982 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98a001a800 con 0x7f98b00faae0 2023-10-12T22:31:16.550 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.538+0000 7f98b6441700 1 -- 172.21.15.131:0/1960707982 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b010b530 con 0x7f98b00faae0 2023-10-12T22:31:16.550 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.538+0000 7f98acff9700 1 -- 172.21.15.131:0/1960707982 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f98a000f070 con 0x7f98b00faae0 2023-10-12T22:31:16.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.539+0000 7f98acff9700 1 --2- 172.21.15.131:0/1960707982 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f989c02fe60 0x7f989c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:16.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.539+0000 7f98acff9700 1 -- 172.21.15.131:0/1960707982 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f98a0040ff0 con 0x7f98b00faae0 2023-10-12T22:31:16.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.539+0000 7f98b6441700 1 -- 172.21.15.131:0/1960707982 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b0045ad0 con 0x7f98b00faae0 2023-10-12T22:31:16.551 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.539+0000 7f98af7fe700 1 --2- 172.21.15.131:0/1960707982 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f989c02fe60 0x7f989c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:16.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.539+0000 7f98af7fe700 1 --2- 172.21.15.131:0/1960707982 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f989c02fe60 0x7f989c032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f98a40096a0 tx=0x7f98a4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:16.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.543+0000 7f98acff9700 1 -- 172.21.15.131:0/1960707982 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98a0016070 con 0x7f98b00faae0 2023-10-12T22:31:16.693 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.692+0000 7f98b6441700 1 -- 172.21.15.131:0/1960707982 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f98b0104a50 con 0x7f98b00faae0 2023-10-12T22:31:16.695 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.694+0000 7f98acff9700 1 -- 172.21.15.131:0/1960707982 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7f98a0024410 con 0x7f98b00faae0 2023-10-12T22:31:16.699 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.696+0000 7f98b6441700 1 -- 172.21.15.131:0/1960707982 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f989c02fe60 msgr2=0x7f989c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:16.700 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.696+0000 7f98b6441700 1 --2- 172.21.15.131:0/1960707982 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f989c02fe60 0x7f989c032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f98a40096a0 tx=0x7f98a4006b40).stop 2023-10-12T22:31:16.700 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.696+0000 7f98b6441700 1 -- 172.21.15.131:0/1960707982 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 msgr2=0x7f98b010aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:16.700 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.696+0000 7f98b6441700 1 --2- 172.21.15.131:0/1960707982 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 0x7f98b010aa30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f98a0006800 tx=0x7f98a0006ee0).stop 2023-10-12T22:31:16.701 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.696+0000 7f98b6441700 1 -- 172.21.15.131:0/1960707982 shutdown_connections 2023-10-12T22:31:16.701 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.696+0000 7f98b6441700 1 --2- 172.21.15.131:0/1960707982 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f989c02fe60 0x7f989c032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:16.702 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.696+0000 7f98b6441700 1 --2- 172.21.15.131:0/1960707982 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98b00faae0 0x7f98b010aa30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:16.702 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.696+0000 7f98b6441700 1 -- 172.21.15.131:0/1960707982 >> 172.21.15.131:0/1960707982 conn(0x7f98b00f6280 msgr2=0x7f98b006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:16.702 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.697+0000 7f98b6441700 1 -- 172.21.15.131:0/1960707982 shutdown_connections 2023-10-12T22:31:16.703 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:16.697+0000 7f98b6441700 1 -- 172.21.15.131:0/1960707982 wait complete. 2023-10-12T22:31:17.062 INFO:teuthology.orchestra.run.smithi131.stderr:Enabling the dashboard module... 2023-10-12T22:31:17.062 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-10-12T22:31:17.399 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.397+0000 7f3232c4f700 1 Processor -- start 2023-10-12T22:31:17.399 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.397+0000 7f3232c4f700 1 -- start start 2023-10-12T22:31:17.400 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.397+0000 7f3232c4f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 0x7f322c100c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:17.400 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.397+0000 7f3232c4f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0fc6a0 con 0x7f322c06fc50 2023-10-12T22:31:17.400 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.397+0000 7f32309eb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 0x7f322c100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:17.400 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.397+0000 7f32309eb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 0x7f322c100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37854/0 (socket says 172.21.15.131:37854) 2023-10-12T22:31:17.401 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.397+0000 7f32309eb700 1 -- 172.21.15.131:0/2931566544 learned_addr learned my addr 172.21.15.131:0/2931566544 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:17.401 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.398+0000 7f32309eb700 1 -- 172.21.15.131:0/2931566544 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c101110 con 0x7f322c06fc50 2023-10-12T22:31:17.401 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.398+0000 7f32309eb700 1 --2- 172.21.15.131:0/2931566544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 0x7f322c100c20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f322000b5f0 tx=0x7f3220006ee0).ready entity=mon.0 client_cookie=a1c01d2790765706 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:17.403 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.398+0000 7f322b7fe700 1 -- 172.21.15.131:0/2931566544 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f322000d600 con 0x7f322c06fc50 2023-10-12T22:31:17.403 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.398+0000 7f322b7fe700 1 -- 172.21.15.131:0/2931566544 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f322000dc10 con 0x7f322c06fc50 2023-10-12T22:31:17.404 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.398+0000 7f322b7fe700 1 -- 172.21.15.131:0/2931566544 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3220011cd0 con 0x7f322c06fc50 2023-10-12T22:31:17.404 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.398+0000 7f3232c4f700 1 -- 172.21.15.131:0/2931566544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 msgr2=0x7f322c100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:17.404 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.398+0000 7f3232c4f700 1 --2- 172.21.15.131:0/2931566544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 0x7f322c100c20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f322000b5f0 tx=0x7f3220006ee0).stop 2023-10-12T22:31:17.404 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.398+0000 7f3232c4f700 1 -- 172.21.15.131:0/2931566544 shutdown_connections 2023-10-12T22:31:17.404 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.398+0000 7f3232c4f700 1 --2- 172.21.15.131:0/2931566544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 0x7f322c100c20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:17.405 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.398+0000 7f3232c4f700 1 -- 172.21.15.131:0/2931566544 >> 172.21.15.131:0/2931566544 conn(0x7f322c0f6280 msgr2=0x7f322c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:17.405 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.399+0000 7f3232c4f700 1 -- 172.21.15.131:0/2931566544 shutdown_connections 2023-10-12T22:31:17.405 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.399+0000 7f3232c4f700 1 -- 172.21.15.131:0/2931566544 wait complete. 2023-10-12T22:31:17.405 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.399+0000 7f3232c4f700 1 Processor -- start 2023-10-12T22:31:17.405 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.399+0000 7f3232c4f700 1 -- start start 2023-10-12T22:31:17.406 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.399+0000 7f3232c4f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 0x7f322c10cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:17.406 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.399+0000 7f3232c4f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c10d1d0 con 0x7f322c06fc50 2023-10-12T22:31:17.406 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.400+0000 7f32309eb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 0x7f322c10cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:17.406 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.400+0000 7f32309eb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 0x7f322c10cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37862/0 (socket says 172.21.15.131:37862) 2023-10-12T22:31:17.406 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.400+0000 7f32309eb700 1 -- 172.21.15.131:0/1559207996 learned_addr learned my addr 172.21.15.131:0/1559207996 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:17.407 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.400+0000 7f32309eb700 1 -- 172.21.15.131:0/1559207996 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322000b200 con 0x7f322c06fc50 2023-10-12T22:31:17.407 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.400+0000 7f32309eb700 1 --2- 172.21.15.131:0/1559207996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 0x7f322c10cce0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f322000bf40 tx=0x7f3220006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:17.407 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.400+0000 7f32297fa700 1 -- 172.21.15.131:0/1559207996 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3220011df0 con 0x7f322c06fc50 2023-10-12T22:31:17.407 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.400+0000 7f3232c4f700 1 -- 172.21.15.131:0/1559207996 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c10d3d0 con 0x7f322c06fc50 2023-10-12T22:31:17.407 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.400+0000 7f32297fa700 1 -- 172.21.15.131:0/1559207996 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f322001b8c0 con 0x7f322c06fc50 2023-10-12T22:31:17.407 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.400+0000 7f32297fa700 1 -- 172.21.15.131:0/1559207996 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f322001a8e0 con 0x7f322c06fc50 2023-10-12T22:31:17.408 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.400+0000 7f3232c4f700 1 -- 172.21.15.131:0/1559207996 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c10d7e0 con 0x7f322c06fc50 2023-10-12T22:31:17.408 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.401+0000 7f32297fa700 1 -- 172.21.15.131:0/1559207996 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f322000f070 con 0x7f322c06fc50 2023-10-12T22:31:17.408 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.401+0000 7f3232c4f700 1 -- 172.21.15.131:0/1559207996 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c045ad0 con 0x7f322c06fc50 2023-10-12T22:31:17.408 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.401+0000 7f32297fa700 1 --2- 172.21.15.131:0/1559207996 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f321002fe60 0x7f3210032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:17.408 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.401+0000 7f32297fa700 1 -- 172.21.15.131:0/1559207996 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f3220040f40 con 0x7f322c06fc50 2023-10-12T22:31:17.409 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.401+0000 7f322bfff700 1 --2- 172.21.15.131:0/1559207996 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f321002fe60 0x7f3210032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:17.409 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.402+0000 7f322bfff700 1 --2- 172.21.15.131:0/1559207996 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f321002fe60 0x7f3210032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f32180096a0 tx=0x7f3218006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:17.409 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.406+0000 7f32297fa700 1 -- 172.21.15.131:0/1559207996 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f322006d2b0 con 0x7f322c06fc50 2023-10-12T22:31:17.594 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.593+0000 7f3232c4f700 1 -- 172.21.15.131:0/1559207996 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f322c101e50 con 0x7f322c06fc50 2023-10-12T22:31:17.696 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.694+0000 7f32297fa700 1 -- 172.21.15.131:0/1559207996 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f3220024790 con 0x7f322c06fc50 2023-10-12T22:31:17.697 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.696+0000 7f32297fa700 1 -- 172.21.15.131:0/1559207996 <== mon.0 v2:172.21.15.131:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7f3220017c40 con 0x7f322c06fc50 2023-10-12T22:31:17.700 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.698+0000 7f3232c4f700 1 -- 172.21.15.131:0/1559207996 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f321002fe60 msgr2=0x7f3210032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:17.700 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.698+0000 7f3232c4f700 1 --2- 172.21.15.131:0/1559207996 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f321002fe60 0x7f3210032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f32180096a0 tx=0x7f3218006b40).stop 2023-10-12T22:31:17.700 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.698+0000 7f3232c4f700 1 -- 172.21.15.131:0/1559207996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 msgr2=0x7f322c10cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:17.701 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.698+0000 7f3232c4f700 1 --2- 172.21.15.131:0/1559207996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 0x7f322c10cce0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f322000bf40 tx=0x7f3220006ee0).stop 2023-10-12T22:31:17.701 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.698+0000 7f3232c4f700 1 -- 172.21.15.131:0/1559207996 shutdown_connections 2023-10-12T22:31:17.701 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.698+0000 7f3232c4f700 1 --2- 172.21.15.131:0/1559207996 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f321002fe60 0x7f3210032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:17.701 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.698+0000 7f3232c4f700 1 --2- 172.21.15.131:0/1559207996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f322c06fc50 0x7f322c10cce0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:17.701 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.698+0000 7f3232c4f700 1 -- 172.21.15.131:0/1559207996 >> 172.21.15.131:0/1559207996 conn(0x7f322c0f6280 msgr2=0x7f322c0f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:17.702 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.698+0000 7f3232c4f700 1 -- 172.21.15.131:0/1559207996 shutdown_connections 2023-10-12T22:31:17.702 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:17.698+0000 7f3232c4f700 1 -- 172.21.15.131:0/1559207996 wait complete. 2023-10-12T22:31:17.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:17 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:16.695953+0000 mon.smithi131 (mon.0) 86 : audit [INF] from='client.? 172.21.15.131:0/1960707982' entity='client.admin' 2023-10-12T22:31:17.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:17 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:17.594427+0000 mon.smithi131 (mon.0) 87 : audit [INF] from='client.? 172.21.15.131:0/1559207996' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-10-12T22:31:18.069 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-10-12T22:31:18.455 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.416+0000 7f65fedcb700 1 Processor -- start 2023-10-12T22:31:18.455 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.416+0000 7f65fedcb700 1 -- start start 2023-10-12T22:31:18.455 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.416+0000 7f65fedcb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 0x7f65f80fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:18.456 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.416+0000 7f65fedcb700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f80fed20 con 0x7f65f806fcd0 2023-10-12T22:31:18.456 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.417+0000 7f65fcb67700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 0x7f65f80fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:18.456 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.417+0000 7f65fcb67700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 0x7f65f80fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37890/0 (socket says 172.21.15.131:37890) 2023-10-12T22:31:18.456 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.417+0000 7f65fcb67700 1 -- 172.21.15.131:0/1391885714 learned_addr learned my addr 172.21.15.131:0/1391885714 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:18.456 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.417+0000 7f65fcb67700 1 -- 172.21.15.131:0/1391885714 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f80fee60 con 0x7f65f806fcd0 2023-10-12T22:31:18.457 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.417+0000 7f65fcb67700 1 --2- 172.21.15.131:0/1391885714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 0x7f65f80fe7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f65ec00b5f0 tx=0x7f65ec006ee0).ready entity=mon.0 client_cookie=333fad54661b0fcb server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:18.457 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.417+0000 7f65f77fe700 1 -- 172.21.15.131:0/1391885714 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65ec00d600 con 0x7f65f806fcd0 2023-10-12T22:31:18.457 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.417+0000 7f65f77fe700 1 -- 172.21.15.131:0/1391885714 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65ec00dc10 con 0x7f65f806fcd0 2023-10-12T22:31:18.457 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.417+0000 7f65f77fe700 1 -- 172.21.15.131:0/1391885714 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65ec011cd0 con 0x7f65f806fcd0 2023-10-12T22:31:18.457 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.418+0000 7f65fedcb700 1 -- 172.21.15.131:0/1391885714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 msgr2=0x7f65f80fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:18.458 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.418+0000 7f65fedcb700 1 --2- 172.21.15.131:0/1391885714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 0x7f65f80fe7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f65ec00b5f0 tx=0x7f65ec006ee0).stop 2023-10-12T22:31:18.458 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.418+0000 7f65fedcb700 1 -- 172.21.15.131:0/1391885714 shutdown_connections 2023-10-12T22:31:18.458 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.418+0000 7f65fedcb700 1 --2- 172.21.15.131:0/1391885714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 0x7f65f80fe7a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:18.458 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.418+0000 7f65fedcb700 1 -- 172.21.15.131:0/1391885714 >> 172.21.15.131:0/1391885714 conn(0x7f65f80f6280 msgr2=0x7f65f80f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:18.458 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.418+0000 7f65fedcb700 1 -- 172.21.15.131:0/1391885714 shutdown_connections 2023-10-12T22:31:18.459 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.418+0000 7f65fedcb700 1 -- 172.21.15.131:0/1391885714 wait complete. 2023-10-12T22:31:18.459 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.418+0000 7f65fedcb700 1 Processor -- start 2023-10-12T22:31:18.459 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.418+0000 7f65fedcb700 1 -- start start 2023-10-12T22:31:18.459 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.419+0000 7f65fedcb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 0x7f65f810a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:18.459 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.419+0000 7f65fedcb700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f810aee0 con 0x7f65f806fcd0 2023-10-12T22:31:18.460 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.419+0000 7f65fcb67700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 0x7f65f810a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:18.460 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.419+0000 7f65fcb67700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 0x7f65f810a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:37900/0 (socket says 172.21.15.131:37900) 2023-10-12T22:31:18.460 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.419+0000 7f65fcb67700 1 -- 172.21.15.131:0/3928994531 learned_addr learned my addr 172.21.15.131:0/3928994531 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:18.460 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.419+0000 7f65fcb67700 1 -- 172.21.15.131:0/3928994531 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec00b200 con 0x7f65f806fcd0 2023-10-12T22:31:18.460 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.419+0000 7f65fcb67700 1 --2- 172.21.15.131:0/3928994531 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 0x7f65f810a9f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f65ec00b1d0 tx=0x7f65ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:18.460 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.419+0000 7f65f57fa700 1 -- 172.21.15.131:0/3928994531 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65ec011df0 con 0x7f65f806fcd0 2023-10-12T22:31:18.461 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.419+0000 7f65f57fa700 1 -- 172.21.15.131:0/3928994531 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65ec01b8c0 con 0x7f65f806fcd0 2023-10-12T22:31:18.461 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.419+0000 7f65f57fa700 1 -- 172.21.15.131:0/3928994531 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65ec01a7b0 con 0x7f65f806fcd0 2023-10-12T22:31:18.461 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.419+0000 7f65fedcb700 1 -- 172.21.15.131:0/3928994531 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f810b0e0 con 0x7f65f806fcd0 2023-10-12T22:31:18.461 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.419+0000 7f65fedcb700 1 -- 172.21.15.131:0/3928994531 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f810b4f0 con 0x7f65f806fcd0 2023-10-12T22:31:18.461 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.420+0000 7f65f57fa700 1 -- 172.21.15.131:0/3928994531 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f65ec00f070 con 0x7f65f806fcd0 2023-10-12T22:31:18.462 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.420+0000 7f65fedcb700 1 -- 172.21.15.131:0/3928994531 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f8045ad0 con 0x7f65f806fcd0 2023-10-12T22:31:18.462 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.420+0000 7f65f57fa700 1 --2- 172.21.15.131:0/3928994531 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f65dc02ff00 0x7f65dc0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:18.462 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.420+0000 7f65f7fff700 1 -- 172.21.15.131:0/3928994531 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f65dc02ff00 msgr2=0x7f65dc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/3456822191 2023-10-12T22:31:18.462 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.420+0000 7f65f57fa700 1 -- 172.21.15.131:0/3928994531 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f65ec041250 con 0x7f65f806fcd0 2023-10-12T22:31:18.462 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.420+0000 7f65f7fff700 1 --2- 172.21.15.131:0/3928994531 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f65dc02ff00 0x7f65dc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-12T22:31:18.463 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.424+0000 7f65f57fa700 1 -- 172.21.15.131:0/3928994531 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65ec025460 con 0x7f65f806fcd0 2023-10-12T22:31:18.622 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.621+0000 7f65f7fff700 1 -- 172.21.15.131:0/3928994531 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f65dc02ff00 msgr2=0x7f65dc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/3456822191 2023-10-12T22:31:18.623 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.621+0000 7f65f7fff700 1 --2- 172.21.15.131:0/3928994531 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f65dc02ff00 0x7f65dc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-12T22:31:18.624 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.623+0000 7f65fedcb700 1 -- 172.21.15.131:0/3928994531 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f65f810c600 con 0x7f65f806fcd0 2023-10-12T22:31:18.626 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.624+0000 7f65f57fa700 1 -- 172.21.15.131:0/3928994531 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7f65ec016070 con 0x7f65f806fcd0 2023-10-12T22:31:18.626 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2023-10-12T22:31:18.626 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 12, 2023-10-12T22:31:18.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "available": true, 2023-10-12T22:31:18.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "active_name": "smithi131.efktma", 2023-10-12T22:31:18.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_standby": 0 2023-10-12T22:31:18.627 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2023-10-12T22:31:18.629 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.625+0000 7f65fedcb700 1 -- 172.21.15.131:0/3928994531 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f65dc02ff00 msgr2=0x7f65dc0323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-12T22:31:18.629 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.626+0000 7f65fedcb700 1 --2- 172.21.15.131:0/3928994531 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f65dc02ff00 0x7f65dc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-12T22:31:18.629 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.626+0000 7f65fedcb700 1 -- 172.21.15.131:0/3928994531 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 msgr2=0x7f65f810a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:18.629 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.626+0000 7f65fedcb700 1 --2- 172.21.15.131:0/3928994531 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 0x7f65f810a9f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f65ec00b1d0 tx=0x7f65ec006ee0).stop 2023-10-12T22:31:18.629 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.626+0000 7f65fedcb700 1 -- 172.21.15.131:0/3928994531 shutdown_connections 2023-10-12T22:31:18.630 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.626+0000 7f65fedcb700 1 --2- 172.21.15.131:0/3928994531 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f65dc02ff00 0x7f65dc0323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-12T22:31:18.630 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.626+0000 7f65fedcb700 1 --2- 172.21.15.131:0/3928994531 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65f806fcd0 0x7f65f810a9f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:18.630 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.626+0000 7f65fedcb700 1 -- 172.21.15.131:0/3928994531 >> 172.21.15.131:0/3928994531 conn(0x7f65f80f6280 msgr2=0x7f65f80f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:18.630 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.626+0000 7f65fedcb700 1 -- 172.21.15.131:0/3928994531 shutdown_connections 2023-10-12T22:31:18.630 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:18.626+0000 7f65fedcb700 1 -- 172.21.15.131:0/3928994531 wait complete. 2023-10-12T22:31:18.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:18 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:17.697286+0000 mon.smithi131 (mon.0) 88 : audit [INF] from='client.? 172.21.15.131:0/1559207996' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-10-12T22:31:18.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:18 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:17.697331+0000 mon.smithi131 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi131.efktma(active, since 7s) 2023-10-12T22:31:18.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:18 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:18.625293+0000 mon.smithi131 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.131:0/3928994531' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-10-12T22:31:18.980 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for the mgr to restart... 2023-10-12T22:31:18.980 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for mgr epoch 12... 2023-10-12T22:31:18.980 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-10-12T22:31:19.343 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.339+0000 7f17a4641700 1 Processor -- start 2023-10-12T22:31:19.344 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.339+0000 7f17a4641700 1 -- start start 2023-10-12T22:31:19.344 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.339+0000 7f17a4641700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 0x7f179c100c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:19.344 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.339+0000 7f17a4641700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c0fc6a0 con 0x7f179c06fc50 2023-10-12T22:31:19.344 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.339+0000 7f17a23dd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 0x7f179c100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:19.345 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.339+0000 7f17a23dd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 0x7f179c100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:34256/0 (socket says 172.21.15.131:34256) 2023-10-12T22:31:19.345 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.339+0000 7f17a23dd700 1 -- 172.21.15.131:0/3486323881 learned_addr learned my addr 172.21.15.131:0/3486323881 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:19.345 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.340+0000 7f17a23dd700 1 -- 172.21.15.131:0/3486323881 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c101110 con 0x7f179c06fc50 2023-10-12T22:31:19.345 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.340+0000 7f17a23dd700 1 --2- 172.21.15.131:0/3486323881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 0x7f179c100c20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f179800b5f0 tx=0x7f1798006ee0).ready entity=mon.0 client_cookie=145cbad8566b229f server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:19.345 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.340+0000 7f17a13db700 1 -- 172.21.15.131:0/3486323881 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f179800d600 con 0x7f179c06fc50 2023-10-12T22:31:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.340+0000 7f17a13db700 1 -- 172.21.15.131:0/3486323881 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f179800dc10 con 0x7f179c06fc50 2023-10-12T22:31:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.340+0000 7f17a13db700 1 -- 172.21.15.131:0/3486323881 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1798011cd0 con 0x7f179c06fc50 2023-10-12T22:31:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.341+0000 7f17a4641700 1 -- 172.21.15.131:0/3486323881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 msgr2=0x7f179c100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.341+0000 7f17a4641700 1 --2- 172.21.15.131:0/3486323881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 0x7f179c100c20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f179800b5f0 tx=0x7f1798006ee0).stop 2023-10-12T22:31:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.341+0000 7f17a4641700 1 -- 172.21.15.131:0/3486323881 shutdown_connections 2023-10-12T22:31:19.347 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.341+0000 7f17a4641700 1 --2- 172.21.15.131:0/3486323881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 0x7f179c100c20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:19.347 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.341+0000 7f17a4641700 1 -- 172.21.15.131:0/3486323881 >> 172.21.15.131:0/3486323881 conn(0x7f179c0f6280 msgr2=0x7f179c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:19.347 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.341+0000 7f17a4641700 1 -- 172.21.15.131:0/3486323881 shutdown_connections 2023-10-12T22:31:19.347 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.341+0000 7f17a4641700 1 -- 172.21.15.131:0/3486323881 wait complete. 2023-10-12T22:31:19.347 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.342+0000 7f17a4641700 1 Processor -- start 2023-10-12T22:31:19.348 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.342+0000 7f17a4641700 1 -- start start 2023-10-12T22:31:19.348 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.342+0000 7f17a4641700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 0x7f179c10cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:19.348 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.342+0000 7f17a4641700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c10d220 con 0x7f179c06fc50 2023-10-12T22:31:19.348 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.342+0000 7f17a23dd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 0x7f179c10cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:19.348 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.342+0000 7f17a23dd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 0x7f179c10cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:34270/0 (socket says 172.21.15.131:34270) 2023-10-12T22:31:19.348 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.342+0000 7f17a23dd700 1 -- 172.21.15.131:0/1815937412 learned_addr learned my addr 172.21.15.131:0/1815937412 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:19.349 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.342+0000 7f17a23dd700 1 -- 172.21.15.131:0/1815937412 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179800b200 con 0x7f179c06fc50 2023-10-12T22:31:19.350 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.343+0000 7f17a23dd700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 0x7f179c10cd30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f179800c010 tx=0x7f1798006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:19.350 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.343+0000 7f178affd700 1 -- 172.21.15.131:0/1815937412 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1798011df0 con 0x7f179c06fc50 2023-10-12T22:31:19.350 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.343+0000 7f17a4641700 1 -- 172.21.15.131:0/1815937412 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f179c10d420 con 0x7f179c06fc50 2023-10-12T22:31:19.350 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.343+0000 7f178affd700 1 -- 172.21.15.131:0/1815937412 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f179801b8c0 con 0x7f179c06fc50 2023-10-12T22:31:19.351 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.343+0000 7f178affd700 1 -- 172.21.15.131:0/1815937412 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f179801a830 con 0x7f179c06fc50 2023-10-12T22:31:19.351 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.343+0000 7f17a4641700 1 -- 172.21.15.131:0/1815937412 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f179c10d830 con 0x7f179c06fc50 2023-10-12T22:31:19.351 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.344+0000 7f178affd700 1 -- 172.21.15.131:0/1815937412 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f179800f070 con 0x7f179c06fc50 2023-10-12T22:31:19.351 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.344+0000 7f178affd700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 0x7f1784032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:19.352 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.344+0000 7f178affd700 1 -- 172.21.15.131:0/1815937412 --> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1784032a30 con 0x7f178402feb0 2023-10-12T22:31:19.352 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.344+0000 7f178affd700 1 -- 172.21.15.131:0/1815937412 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1798040f70 con 0x7f179c06fc50 2023-10-12T22:31:19.352 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.344+0000 7f17a1bdc700 1 -- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 msgr2=0x7f1784032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/3456822191 2023-10-12T22:31:19.353 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.344+0000 7f17a1bdc700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 0x7f1784032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-10-12T22:31:19.546 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.545+0000 7f17a1bdc700 1 -- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 msgr2=0x7f1784032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/3456822191 2023-10-12T22:31:19.546 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.545+0000 7f17a1bdc700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 0x7f1784032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-10-12T22:31:19.947 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.946+0000 7f17a1bdc700 1 -- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 msgr2=0x7f1784032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/3456822191 2023-10-12T22:31:19.947 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:19.946+0000 7f17a1bdc700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 0x7f1784032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-10-12T22:31:20.748 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:20.747+0000 7f17a1bdc700 1 -- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 msgr2=0x7f1784032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/3456822191 2023-10-12T22:31:20.748 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:20.747+0000 7f17a1bdc700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 0x7f1784032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-10-12T22:31:22.350 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:22.348+0000 7f17a1bdc700 1 -- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 msgr2=0x7f1784032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.131:6800/3456822191 2023-10-12T22:31:22.350 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:22.348+0000 7f17a1bdc700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 0x7f1784032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-10-12T22:31:22.792 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:22.790+0000 7f178affd700 1 -- 172.21.15.131:0/1815937412 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f1798010dd0 con 0x7f179c06fc50 2023-10-12T22:31:22.792 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:22.790+0000 7f178affd700 1 -- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 msgr2=0x7f1784032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-12T22:31:22.793 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:22.790+0000 7f178affd700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 0x7f1784032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-12T22:31:22.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:22.739793+0000 mon.smithi131 (mon.0) 91 : cluster [INF] Active manager daemon smithi131.efktma restarted 2023-10-12T22:31:22.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:22.740238+0000 mon.smithi131 (mon.0) 92 : cluster [INF] Activating manager daemon smithi131.efktma 2023-10-12T22:31:22.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:22.741910+0000 mon.smithi131 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-10-12T22:31:23.799 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.797+0000 7f178affd700 1 -- 172.21.15.131:0/1815937412 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f1798041f80 con 0x7f179c06fc50 2023-10-12T22:31:23.799 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.798+0000 7f178affd700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1784033b40 0x7f1784035f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:23.800 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.798+0000 7f178affd700 1 -- 172.21.15.131:0/1815937412 --> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1784032a30 con 0x7f1784033b40 2023-10-12T22:31:23.800 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.798+0000 7f17a1bdc700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1784033b40 0x7f1784035f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:23.800 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.799+0000 7f17a1bdc700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1784033b40 0x7f1784035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f178c006fd0 tx=0x7f178c006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:23.801 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.799+0000 7f178affd700 1 -- 172.21.15.131:0/1815937412 <== mgr.14162 v2:172.21.15.131:6800/3705786429 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f1784032a30 con 0x7f1784033b40 2023-10-12T22:31:23.808 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.807+0000 7f17a4641700 1 -- 172.21.15.131:0/1815937412 --> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f179c059100 con 0x7f1784033b40 2023-10-12T22:31:23.808 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.807+0000 7f178affd700 1 -- 172.21.15.131:0/1815937412 <== mgr.14162 v2:172.21.15.131:6800/3705786429 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f179c059100 con 0x7f1784033b40 2023-10-12T22:31:23.809 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2023-10-12T22:31:23.809 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2023-10-12T22:31:23.810 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "initialized": true 2023-10-12T22:31:23.811 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2023-10-12T22:31:23.811 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.807+0000 7f17a4641700 1 -- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1784033b40 msgr2=0x7f1784035f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:23.811 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.807+0000 7f17a4641700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1784033b40 0x7f1784035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f178c006fd0 tx=0x7f178c006cf0).stop 2023-10-12T22:31:23.811 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.807+0000 7f17a4641700 1 -- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 msgr2=0x7f179c10cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:23.811 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.807+0000 7f17a4641700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 0x7f179c10cd30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f179800c010 tx=0x7f1798006ee0).stop 2023-10-12T22:31:23.812 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.808+0000 7f17a4641700 1 -- 172.21.15.131:0/1815937412 shutdown_connections 2023-10-12T22:31:23.812 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.808+0000 7f17a4641700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1784033b40 0x7f1784035f30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:23.812 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.808+0000 7f17a4641700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f179c06fc50 0x7f179c10cd30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:23.812 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.808+0000 7f17a4641700 1 --2- 172.21.15.131:0/1815937412 >> [v2:172.21.15.131:6800/3456822191,v1:172.21.15.131:6801/3456822191] conn(0x7f178402feb0 0x7f1784032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-10-12T22:31:23.812 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.808+0000 7f17a4641700 1 -- 172.21.15.131:0/1815937412 >> 172.21.15.131:0/1815937412 conn(0x7f179c0f6280 msgr2=0x7f179c0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:23.812 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.808+0000 7f17a4641700 1 -- 172.21.15.131:0/1815937412 shutdown_connections 2023-10-12T22:31:23.813 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:23.808+0000 7f17a4641700 1 -- 172.21.15.131:0/1815937412 wait complete. 2023-10-12T22:31:23.959 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:22.792686+0000 mon.smithi131 (mon.0) 94 : cluster [DBG] mgrmap e13: smithi131.efktma(active, starting, since 0.0525369s) 2023-10-12T22:31:23.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:22.794579+0000 mon.smithi131 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2023-10-12T22:31:23.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:22.795199+0000 mon.smithi131 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mgr metadata", "who": "smithi131.efktma", "id": "smithi131.efktma"}]: dispatch 2023-10-12T22:31:23.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:22.796060+0000 mon.smithi131 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-12T22:31:23.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:22.796268+0000 mon.smithi131 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-12T22:31:23.961 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:22.798211+0000 mon.smithi131 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-12T22:31:23.961 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:22.811632+0000 mon.smithi131 (mon.0) 100 : cluster [INF] Manager daemon smithi131.efktma is now available 2023-10-12T22:31:23.961 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:22.910330+0000 mon.smithi131 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-12T22:31:23.961 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:22.912665+0000 mon.smithi131 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-12T22:31:23.961 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:22.927937+0000 mon.smithi131 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/mirror_snapshot_schedule"}]: dispatch 2023-10-12T22:31:23.961 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:22.931428+0000 mon.smithi131 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/trash_purge_schedule"}]: dispatch 2023-10-12T22:31:23.962 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:23.714396+0000 mon.smithi131 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:31:24.161 INFO:teuthology.orchestra.run.smithi131.stderr:mgr epoch 12 is available 2023-10-12T22:31:24.161 INFO:teuthology.orchestra.run.smithi131.stderr:Generating a dashboard self-signed certificate... 2023-10-12T22:31:24.162 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-10-12T22:31:24.519 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.517+0000 7f1e97016700 1 Processor -- start 2023-10-12T22:31:24.519 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.517+0000 7f1e97016700 1 -- start start 2023-10-12T22:31:24.520 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.517+0000 7f1e97016700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 0x7f1e90100ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:24.520 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.517+0000 7f1e97016700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e900fc4a0 con 0x7f1e9006fbd0 2023-10-12T22:31:24.523 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.518+0000 7f1e96014700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 0x7f1e90100ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:24.523 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.518+0000 7f1e96014700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 0x7f1e90100ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:34334/0 (socket says 172.21.15.131:34334) 2023-10-12T22:31:24.523 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.518+0000 7f1e96014700 1 -- 172.21.15.131:0/177664574 learned_addr learned my addr 172.21.15.131:0/177664574 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:24.523 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.518+0000 7f1e96014700 1 -- 172.21.15.131:0/177664574 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e90100fa0 con 0x7f1e9006fbd0 2023-10-12T22:31:24.523 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.518+0000 7f1e96014700 1 --2- 172.21.15.131:0/177664574 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 0x7f1e90100ab0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f1e8400b5f0 tx=0x7f1e84006ee0).ready entity=mon.0 client_cookie=ebb02e00487b0fe4 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:24.524 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.518+0000 7f1e95012700 1 -- 172.21.15.131:0/177664574 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e8400d600 con 0x7f1e9006fbd0 2023-10-12T22:31:24.524 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.518+0000 7f1e95012700 1 -- 172.21.15.131:0/177664574 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e8400dc10 con 0x7f1e9006fbd0 2023-10-12T22:31:24.524 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.518+0000 7f1e95012700 1 -- 172.21.15.131:0/177664574 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e84011cd0 con 0x7f1e9006fbd0 2023-10-12T22:31:24.524 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.519+0000 7f1e97016700 1 -- 172.21.15.131:0/177664574 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 msgr2=0x7f1e90100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:24.524 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.519+0000 7f1e97016700 1 --2- 172.21.15.131:0/177664574 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 0x7f1e90100ab0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f1e8400b5f0 tx=0x7f1e84006ee0).stop 2023-10-12T22:31:24.526 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.519+0000 7f1e97016700 1 -- 172.21.15.131:0/177664574 shutdown_connections 2023-10-12T22:31:24.526 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.519+0000 7f1e97016700 1 --2- 172.21.15.131:0/177664574 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 0x7f1e90100ab0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:24.526 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.519+0000 7f1e97016700 1 -- 172.21.15.131:0/177664574 >> 172.21.15.131:0/177664574 conn(0x7f1e900f6130 msgr2=0x7f1e900f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:24.527 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.519+0000 7f1e97016700 1 -- 172.21.15.131:0/177664574 shutdown_connections 2023-10-12T22:31:24.527 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.519+0000 7f1e97016700 1 -- 172.21.15.131:0/177664574 wait complete. 2023-10-12T22:31:24.527 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e97016700 1 Processor -- start 2023-10-12T22:31:24.527 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e97016700 1 -- start start 2023-10-12T22:31:24.527 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e97016700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 0x7f1e9010c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:24.527 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e97016700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9010cee0 con 0x7f1e9006fbd0 2023-10-12T22:31:24.528 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e96014700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 0x7f1e9010c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:24.528 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e96014700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 0x7f1e9010c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:34350/0 (socket says 172.21.15.131:34350) 2023-10-12T22:31:24.528 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e96014700 1 -- 172.21.15.131:0/3743542188 learned_addr learned my addr 172.21.15.131:0/3743542188 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:24.528 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e96014700 1 -- 172.21.15.131:0/3743542188 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8400b200 con 0x7f1e9006fbd0 2023-10-12T22:31:24.528 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e96014700 1 --2- 172.21.15.131:0/3743542188 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 0x7f1e9010c9f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f1e84006800 tx=0x7f1e84006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:24.529 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e82ffd700 1 -- 172.21.15.131:0/3743542188 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e84011c90 con 0x7f1e9006fbd0 2023-10-12T22:31:24.529 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e82ffd700 1 -- 172.21.15.131:0/3743542188 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e8401b8c0 con 0x7f1e9006fbd0 2023-10-12T22:31:24.529 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e97016700 1 -- 172.21.15.131:0/3743542188 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9010d0e0 con 0x7f1e9006fbd0 2023-10-12T22:31:24.529 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e82ffd700 1 -- 172.21.15.131:0/3743542188 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e8401a880 con 0x7f1e9006fbd0 2023-10-12T22:31:24.529 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.520+0000 7f1e97016700 1 -- 172.21.15.131:0/3743542188 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9010d4f0 con 0x7f1e9006fbd0 2023-10-12T22:31:24.529 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.521+0000 7f1e82ffd700 1 -- 172.21.15.131:0/3743542188 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f1e8400f070 con 0x7f1e9006fbd0 2023-10-12T22:31:24.530 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.521+0000 7f1e97016700 1 -- 172.21.15.131:0/3743542188 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e90045ad0 con 0x7f1e9006fbd0 2023-10-12T22:31:24.530 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.522+0000 7f1e82ffd700 1 --2- 172.21.15.131:0/3743542188 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1e7802fd70 0x7f1e78032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:24.530 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.522+0000 7f1e82ffd700 1 -- 172.21.15.131:0/3743542188 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e840409e0 con 0x7f1e9006fbd0 2023-10-12T22:31:24.531 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.522+0000 7f1e95813700 1 --2- 172.21.15.131:0/3743542188 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1e7802fd70 0x7f1e78032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:24.531 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.522+0000 7f1e95813700 1 --2- 172.21.15.131:0/3743542188 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1e7802fd70 0x7f1e78032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1e8c0096a0 tx=0x7f1e8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:24.531 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.526+0000 7f1e82ffd700 1 -- 172.21.15.131:0/3743542188 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e8401b410 con 0x7f1e9006fbd0 2023-10-12T22:31:24.689 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.688+0000 7f1e97016700 1 -- 172.21.15.131:0/3743542188 --> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e9010ddd0 con 0x7f1e7802fd70 2023-10-12T22:31:24.771 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.769+0000 7f1e82ffd700 1 -- 172.21.15.131:0/3743542188 <== mgr.14162 v2:172.21.15.131:6800/3705786429 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f1e9010ddd0 con 0x7f1e7802fd70 2023-10-12T22:31:24.771 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Self-signed certificate created 2023-10-12T22:31:24.773 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.771+0000 7f1e97016700 1 -- 172.21.15.131:0/3743542188 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1e7802fd70 msgr2=0x7f1e78032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:24.773 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.771+0000 7f1e97016700 1 --2- 172.21.15.131:0/3743542188 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1e7802fd70 0x7f1e78032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1e8c0096a0 tx=0x7f1e8c006b40).stop 2023-10-12T22:31:24.774 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.771+0000 7f1e97016700 1 -- 172.21.15.131:0/3743542188 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 msgr2=0x7f1e9010c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:24.774 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.771+0000 7f1e97016700 1 --2- 172.21.15.131:0/3743542188 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 0x7f1e9010c9f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f1e84006800 tx=0x7f1e84006ee0).stop 2023-10-12T22:31:24.774 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.771+0000 7f1e97016700 1 -- 172.21.15.131:0/3743542188 shutdown_connections 2023-10-12T22:31:24.774 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.771+0000 7f1e97016700 1 --2- 172.21.15.131:0/3743542188 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1e7802fd70 0x7f1e78032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:24.774 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.771+0000 7f1e97016700 1 --2- 172.21.15.131:0/3743542188 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1e9006fbd0 0x7f1e9010c9f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:24.775 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.771+0000 7f1e97016700 1 -- 172.21.15.131:0/3743542188 >> 172.21.15.131:0/3743542188 conn(0x7f1e900f6130 msgr2=0x7f1e900f7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:24.775 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.772+0000 7f1e97016700 1 -- 172.21.15.131:0/3743542188 shutdown_connections 2023-10-12T22:31:24.775 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:24.772+0000 7f1e97016700 1 -- 172.21.15.131:0/3743542188 wait complete. 2023-10-12T22:31:25.126 INFO:teuthology.orchestra.run.smithi131.stderr:Creating initial admin user... 2023-10-12T22:31:25.127 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz3_5th5:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-10-12T22:31:25.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:23.800431+0000 mgr.smithi131.efktma (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-10-12T22:31:25.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:23.801000+0000 mon.smithi131 (mon.0) 106 : cluster [DBG] mgrmap e14: smithi131.efktma(active, since 1.06085s) 2023-10-12T22:31:25.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:23.808574+0000 mgr.smithi131.efktma (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-10-12T22:31:25.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:24.690154+0000 mgr.smithi131.efktma (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:31:25.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:24.768148+0000 mon.smithi131 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:31:25.217 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:24.770613+0000 mon.smithi131 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:31:25.469 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.467+0000 7ffa2087c700 1 Processor -- start 2023-10-12T22:31:25.470 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.467+0000 7ffa2087c700 1 -- start start 2023-10-12T22:31:25.470 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.467+0000 7ffa2087c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 0x7ffa1c070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:25.470 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.468+0000 7ffa2087c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c0720f0 con 0x7ffa1c071cd0 2023-10-12T22:31:25.470 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.468+0000 7ffa1b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 0x7ffa1c070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:25.471 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.468+0000 7ffa1b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 0x7ffa1c070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:34360/0 (socket says 172.21.15.131:34360) 2023-10-12T22:31:25.473 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.468+0000 7ffa1b7fe700 1 -- 172.21.15.131:0/3312340439 learned_addr learned my addr 172.21.15.131:0/3312340439 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:25.474 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.468+0000 7ffa1b7fe700 1 -- 172.21.15.131:0/3312340439 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c070760 con 0x7ffa1c071cd0 2023-10-12T22:31:25.474 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.468+0000 7ffa1b7fe700 1 --2- 172.21.15.131:0/3312340439 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 0x7ffa1c070270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7ffa0c00b5f0 tx=0x7ffa0c006ee0).ready entity=mon.0 client_cookie=6f1c5b21a0d63666 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:25.474 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.468+0000 7ffa1a7fc700 1 -- 172.21.15.131:0/3312340439 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa0c00d600 con 0x7ffa1c071cd0 2023-10-12T22:31:25.474 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.468+0000 7ffa1a7fc700 1 -- 172.21.15.131:0/3312340439 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa0c00dc10 con 0x7ffa1c071cd0 2023-10-12T22:31:25.475 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.468+0000 7ffa1a7fc700 1 -- 172.21.15.131:0/3312340439 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa0c011cd0 con 0x7ffa1c071cd0 2023-10-12T22:31:25.475 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.469+0000 7ffa2087c700 1 -- 172.21.15.131:0/3312340439 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 msgr2=0x7ffa1c070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:25.475 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.469+0000 7ffa2087c700 1 --2- 172.21.15.131:0/3312340439 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 0x7ffa1c070270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7ffa0c00b5f0 tx=0x7ffa0c006ee0).stop 2023-10-12T22:31:25.475 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.469+0000 7ffa2087c700 1 -- 172.21.15.131:0/3312340439 shutdown_connections 2023-10-12T22:31:25.475 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.469+0000 7ffa2087c700 1 --2- 172.21.15.131:0/3312340439 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 0x7ffa1c070270 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:25.475 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.469+0000 7ffa2087c700 1 -- 172.21.15.131:0/3312340439 >> 172.21.15.131:0/3312340439 conn(0x7ffa1c0f71d0 msgr2=0x7ffa1c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:25.476 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.469+0000 7ffa2087c700 1 -- 172.21.15.131:0/3312340439 shutdown_connections 2023-10-12T22:31:25.476 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.469+0000 7ffa2087c700 1 -- 172.21.15.131:0/3312340439 wait complete. 2023-10-12T22:31:25.476 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.470+0000 7ffa2087c700 1 Processor -- start 2023-10-12T22:31:25.476 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.470+0000 7ffa2087c700 1 -- start start 2023-10-12T22:31:25.476 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.470+0000 7ffa2087c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 0x7ffa1c10c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:25.477 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.470+0000 7ffa2087c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c10cee0 con 0x7ffa1c071cd0 2023-10-12T22:31:25.477 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.470+0000 7ffa1b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 0x7ffa1c10c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:25.477 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.470+0000 7ffa1b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 0x7ffa1c10c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:34362/0 (socket says 172.21.15.131:34362) 2023-10-12T22:31:25.477 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.470+0000 7ffa1b7fe700 1 -- 172.21.15.131:0/111150096 learned_addr learned my addr 172.21.15.131:0/111150096 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:25.477 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.470+0000 7ffa1b7fe700 1 -- 172.21.15.131:0/111150096 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c00b200 con 0x7ffa1c071cd0 2023-10-12T22:31:25.478 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.470+0000 7ffa1b7fe700 1 --2- 172.21.15.131:0/111150096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 0x7ffa1c10c9f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ffa0c00b1d0 tx=0x7ffa0c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:25.478 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.471+0000 7ffa13fff700 1 -- 172.21.15.131:0/111150096 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa0c011df0 con 0x7ffa1c071cd0 2023-10-12T22:31:25.478 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.471+0000 7ffa13fff700 1 -- 172.21.15.131:0/111150096 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa0c01b8c0 con 0x7ffa1c071cd0 2023-10-12T22:31:25.478 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.471+0000 7ffa2087c700 1 -- 172.21.15.131:0/111150096 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1c10d0e0 con 0x7ffa1c071cd0 2023-10-12T22:31:25.478 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.471+0000 7ffa13fff700 1 -- 172.21.15.131:0/111150096 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa0c01a830 con 0x7ffa1c071cd0 2023-10-12T22:31:25.479 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.471+0000 7ffa2087c700 1 -- 172.21.15.131:0/111150096 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1c10d4f0 con 0x7ffa1c071cd0 2023-10-12T22:31:25.479 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.471+0000 7ffa13fff700 1 -- 172.21.15.131:0/111150096 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ffa0c00f070 con 0x7ffa1c071cd0 2023-10-12T22:31:25.479 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.471+0000 7ffa13fff700 1 --2- 172.21.15.131:0/111150096 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff9fc02ff00 0x7ff9fc0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:25.479 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.472+0000 7ffa13fff700 1 -- 172.21.15.131:0/111150096 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa0c0410e0 con 0x7ffa1c071cd0 2023-10-12T22:31:25.479 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.472+0000 7ffa1affd700 1 --2- 172.21.15.131:0/111150096 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff9fc02ff00 0x7ff9fc0323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:25.480 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.472+0000 7ffa2087c700 1 -- 172.21.15.131:0/111150096 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1c059100 con 0x7ffa1c071cd0 2023-10-12T22:31:25.480 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.472+0000 7ffa1affd700 1 --2- 172.21.15.131:0/111150096 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff9fc02ff00 0x7ff9fc0323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ffa040096a0 tx=0x7ffa04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:25.480 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.476+0000 7ffa13fff700 1 -- 172.21.15.131:0/111150096 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa0c016070 con 0x7ffa1c071cd0 2023-10-12T22:31:25.642 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.641+0000 7ffa2087c700 1 -- 172.21.15.131:0/111150096 --> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ffa1c045ad0 con 0x7ff9fc02ff00 2023-10-12T22:31:25.862 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.861+0000 7ffa13fff700 1 -- 172.21.15.131:0/111150096 <== mgr.14162 v2:172.21.15.131:6800/3705786429 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7ffa1c045ad0 con 0x7ff9fc02ff00 2023-10-12T22:31:25.862 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$fbNNVwL/ZB3LL3BLoXJH6eph9veFeDIhKR93vFtSOHF7qj/wl0Aae", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1697149885, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-10-12T22:31:25.864 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.863+0000 7ffa2087c700 1 -- 172.21.15.131:0/111150096 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff9fc02ff00 msgr2=0x7ff9fc0323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:25.865 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.863+0000 7ffa2087c700 1 --2- 172.21.15.131:0/111150096 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff9fc02ff00 0x7ff9fc0323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ffa040096a0 tx=0x7ffa04006b40).stop 2023-10-12T22:31:25.865 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.863+0000 7ffa2087c700 1 -- 172.21.15.131:0/111150096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 msgr2=0x7ffa1c10c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:25.865 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.863+0000 7ffa2087c700 1 --2- 172.21.15.131:0/111150096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 0x7ffa1c10c9f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ffa0c00b1d0 tx=0x7ffa0c006ee0).stop 2023-10-12T22:31:25.865 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.863+0000 7ffa2087c700 1 -- 172.21.15.131:0/111150096 shutdown_connections 2023-10-12T22:31:25.865 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.863+0000 7ffa2087c700 1 --2- 172.21.15.131:0/111150096 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff9fc02ff00 0x7ff9fc0323c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:25.866 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.863+0000 7ffa2087c700 1 --2- 172.21.15.131:0/111150096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa1c071cd0 0x7ffa1c10c9f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:25.866 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.863+0000 7ffa2087c700 1 -- 172.21.15.131:0/111150096 >> 172.21.15.131:0/111150096 conn(0x7ffa1c0f71d0 msgr2=0x7ffa1c0f8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:25.866 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.863+0000 7ffa2087c700 1 -- 172.21.15.131:0/111150096 shutdown_connections 2023-10-12T22:31:25.866 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:25.863+0000 7ffa2087c700 1 -- 172.21.15.131:0/111150096 wait complete. 2023-10-12T22:31:26.137 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:25 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:24.805747+0000 mon.smithi131 (mon.0) 109 : cluster [DBG] mgrmap e15: smithi131.efktma(active, since 2s) 2023-10-12T22:31:26.137 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:25 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:25.642945+0000 mgr.smithi131.efktma (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:31:26.210 INFO:teuthology.orchestra.run.smithi131.stderr:Fetching dashboard port number... 2023-10-12T22:31:26.211 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpm5xh9jbd:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpdcj2a7b8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-10-12T22:31:26.552 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.548+0000 7fe601dde700 1 Processor -- start 2023-10-12T22:31:26.553 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.548+0000 7fe601dde700 1 -- start start 2023-10-12T22:31:26.553 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.548+0000 7fe601dde700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 0x7fe5fc0fe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:26.553 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.548+0000 7fe601dde700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0fed80 con 0x7fe5fc06fc70 2023-10-12T22:31:26.553 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.549+0000 7fe5fb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 0x7fe5fc0fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:26.553 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.549+0000 7fe5fb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 0x7fe5fc0fe800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:34368/0 (socket says 172.21.15.131:34368) 2023-10-12T22:31:26.554 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.549+0000 7fe5fb7fe700 1 -- 172.21.15.131:0/113502118 learned_addr learned my addr 172.21.15.131:0/113502118 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:26.554 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.549+0000 7fe5fb7fe700 1 -- 172.21.15.131:0/113502118 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc0feec0 con 0x7fe5fc06fc70 2023-10-12T22:31:26.554 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.549+0000 7fe5fb7fe700 1 --2- 172.21.15.131:0/113502118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 0x7fe5fc0fe800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fe5f000b5f0 tx=0x7fe5f0006ee0).ready entity=mon.0 client_cookie=b82005289be80143 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:26.554 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.549+0000 7fe5fa7fc700 1 -- 172.21.15.131:0/113502118 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5f000d600 con 0x7fe5fc06fc70 2023-10-12T22:31:26.554 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.549+0000 7fe5fa7fc700 1 -- 172.21.15.131:0/113502118 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5f000dc10 con 0x7fe5fc06fc70 2023-10-12T22:31:26.555 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.549+0000 7fe5fa7fc700 1 -- 172.21.15.131:0/113502118 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5f0011cd0 con 0x7fe5fc06fc70 2023-10-12T22:31:26.555 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.550+0000 7fe601dde700 1 -- 172.21.15.131:0/113502118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 msgr2=0x7fe5fc0fe800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:26.555 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.550+0000 7fe601dde700 1 --2- 172.21.15.131:0/113502118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 0x7fe5fc0fe800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fe5f000b5f0 tx=0x7fe5f0006ee0).stop 2023-10-12T22:31:26.555 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.550+0000 7fe601dde700 1 -- 172.21.15.131:0/113502118 shutdown_connections 2023-10-12T22:31:26.555 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.550+0000 7fe601dde700 1 --2- 172.21.15.131:0/113502118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 0x7fe5fc0fe800 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:26.556 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.550+0000 7fe601dde700 1 -- 172.21.15.131:0/113502118 >> 172.21.15.131:0/113502118 conn(0x7fe5fc0f6280 msgr2=0x7fe5fc0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:26.556 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.550+0000 7fe601dde700 1 -- 172.21.15.131:0/113502118 shutdown_connections 2023-10-12T22:31:26.556 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.550+0000 7fe601dde700 1 -- 172.21.15.131:0/113502118 wait complete. 2023-10-12T22:31:26.556 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.550+0000 7fe601dde700 1 Processor -- start 2023-10-12T22:31:26.556 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe601dde700 1 -- start start 2023-10-12T22:31:26.556 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe601dde700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 0x7fe5fc10a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:26.557 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe601dde700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc10aed0 con 0x7fe5fc06fc70 2023-10-12T22:31:26.557 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe5fb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 0x7fe5fc10a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:26.557 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe5fb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 0x7fe5fc10a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:34376/0 (socket says 172.21.15.131:34376) 2023-10-12T22:31:26.557 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe5fb7fe700 1 -- 172.21.15.131:0/1472322002 learned_addr learned my addr 172.21.15.131:0/1472322002 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:26.557 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe5fb7fe700 1 -- 172.21.15.131:0/1472322002 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f000b200 con 0x7fe5fc06fc70 2023-10-12T22:31:26.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe5fb7fe700 1 --2- 172.21.15.131:0/1472322002 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 0x7fe5fc10a9e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fe5f0006cb0 tx=0x7fe5f0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:26.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe5e3fff700 1 -- 172.21.15.131:0/1472322002 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5f0011df0 con 0x7fe5fc06fc70 2023-10-12T22:31:26.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe5e3fff700 1 -- 172.21.15.131:0/1472322002 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5f001b8c0 con 0x7fe5fc06fc70 2023-10-12T22:31:26.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe5e3fff700 1 -- 172.21.15.131:0/1472322002 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5f001a930 con 0x7fe5fc06fc70 2023-10-12T22:31:26.559 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe601dde700 1 -- 172.21.15.131:0/1472322002 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5fc10b0d0 con 0x7fe5fc06fc70 2023-10-12T22:31:26.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.551+0000 7fe601dde700 1 -- 172.21.15.131:0/1472322002 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5fc10b4e0 con 0x7fe5fc06fc70 2023-10-12T22:31:26.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.552+0000 7fe5e3fff700 1 -- 172.21.15.131:0/1472322002 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fe5f000f070 con 0x7fe5fc06fc70 2023-10-12T22:31:26.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.552+0000 7fe5e3fff700 1 --2- 172.21.15.131:0/1472322002 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe5e802feb0 0x7fe5e8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:26.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.552+0000 7fe5e3fff700 1 -- 172.21.15.131:0/1472322002 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe5f0041070 con 0x7fe5fc06fc70 2023-10-12T22:31:26.560 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.552+0000 7fe601dde700 1 -- 172.21.15.131:0/1472322002 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5fc045ad0 con 0x7fe5fc06fc70 2023-10-12T22:31:26.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.553+0000 7fe5faffd700 1 --2- 172.21.15.131:0/1472322002 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe5e802feb0 0x7fe5e8032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:26.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.553+0000 7fe5faffd700 1 --2- 172.21.15.131:0/1472322002 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe5e802feb0 0x7fe5e8032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe5ec0096a0 tx=0x7fe5ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:26.561 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.557+0000 7fe5e3fff700 1 -- 172.21.15.131:0/1472322002 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5f0016070 con 0x7fe5fc06fc70 2023-10-12T22:31:26.708 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.706+0000 7fe601dde700 1 -- 172.21.15.131:0/1472322002 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fe5fc104970 con 0x7fe5fc06fc70 2023-10-12T22:31:26.708 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.706+0000 7fe5e3fff700 1 -- 172.21.15.131:0/1472322002 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7fe5f0019950 con 0x7fe5fc06fc70 2023-10-12T22:31:26.708 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 8443 2023-10-12T22:31:26.710 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.708+0000 7fe601dde700 1 -- 172.21.15.131:0/1472322002 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe5e802feb0 msgr2=0x7fe5e8032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:26.710 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.708+0000 7fe601dde700 1 --2- 172.21.15.131:0/1472322002 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe5e802feb0 0x7fe5e8032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe5ec0096a0 tx=0x7fe5ec006b40).stop 2023-10-12T22:31:26.711 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.708+0000 7fe601dde700 1 -- 172.21.15.131:0/1472322002 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 msgr2=0x7fe5fc10a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:26.711 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.708+0000 7fe601dde700 1 --2- 172.21.15.131:0/1472322002 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 0x7fe5fc10a9e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fe5f0006cb0 tx=0x7fe5f0006ee0).stop 2023-10-12T22:31:26.711 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.708+0000 7fe601dde700 1 -- 172.21.15.131:0/1472322002 shutdown_connections 2023-10-12T22:31:26.711 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.708+0000 7fe601dde700 1 --2- 172.21.15.131:0/1472322002 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe5e802feb0 0x7fe5e8032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:26.712 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.708+0000 7fe601dde700 1 --2- 172.21.15.131:0/1472322002 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5fc06fc70 0x7fe5fc10a9e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:26.712 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.708+0000 7fe601dde700 1 -- 172.21.15.131:0/1472322002 >> 172.21.15.131:0/1472322002 conn(0x7fe5fc0f6280 msgr2=0x7fe5fc0f7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:26.712 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.708+0000 7fe601dde700 1 -- 172.21.15.131:0/1472322002 shutdown_connections 2023-10-12T22:31:26.713 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2023-10-12T22:31:26.708+0000 7fe601dde700 1 -- 172.21.15.131:0/1472322002 wait complete. 2023-10-12T22:31:26.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:26 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:25.862152+0000 mon.smithi131 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:31:26.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:26 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:26.707815+0000 mon.smithi131 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.131:0/1472322002' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-10-12T22:31:27.051 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-enabled firewalld.service 2023-10-12T22:31:27.059 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: disabled 2023-10-12T22:31:27.060 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: systemctl is-active firewalld.service 2023-10-12T22:31:27.066 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: inactive 2023-10-12T22:31:27.067 INFO:teuthology.orchestra.run.smithi131.stderr:firewalld.service is not enabled 2023-10-12T22:31:27.067 INFO:teuthology.orchestra.run.smithi131.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-10-12T22:31:27.070 INFO:teuthology.orchestra.run.smithi131.stderr:Ceph Dashboard is now available at: 2023-10-12T22:31:27.070 INFO:teuthology.orchestra.run.smithi131.stderr: 2023-10-12T22:31:27.071 INFO:teuthology.orchestra.run.smithi131.stderr: URL: https://smithi131.front.sepia.ceph.com:8443/ 2023-10-12T22:31:27.071 INFO:teuthology.orchestra.run.smithi131.stderr: User: admin 2023-10-12T22:31:27.071 INFO:teuthology.orchestra.run.smithi131.stderr: Password: q35rr6iqd1 2023-10-12T22:31:27.071 INFO:teuthology.orchestra.run.smithi131.stderr: 2023-10-12T22:31:27.071 INFO:teuthology.orchestra.run.smithi131.stderr:You can access the Ceph CLI with: 2023-10-12T22:31:27.072 INFO:teuthology.orchestra.run.smithi131.stderr: 2023-10-12T22:31:27.072 INFO:teuthology.orchestra.run.smithi131.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-10-12T22:31:27.072 INFO:teuthology.orchestra.run.smithi131.stderr: 2023-10-12T22:31:27.072 INFO:teuthology.orchestra.run.smithi131.stderr:Please consider enabling telemetry to help improve Ceph: 2023-10-12T22:31:27.072 INFO:teuthology.orchestra.run.smithi131.stderr: 2023-10-12T22:31:27.073 INFO:teuthology.orchestra.run.smithi131.stderr: ceph telemetry on 2023-10-12T22:31:27.073 INFO:teuthology.orchestra.run.smithi131.stderr: 2023-10-12T22:31:27.073 INFO:teuthology.orchestra.run.smithi131.stderr:For more information see: 2023-10-12T22:31:27.073 INFO:teuthology.orchestra.run.smithi131.stderr: 2023-10-12T22:31:27.073 INFO:teuthology.orchestra.run.smithi131.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-10-12T22:31:27.074 INFO:teuthology.orchestra.run.smithi131.stderr: 2023-10-12T22:31:27.074 INFO:teuthology.orchestra.run.smithi131.stderr:Bootstrap complete. 2023-10-12T22:31:27.074 INFO:teuthology.orchestra.run.smithi131.stderr:Releasing lock 140242855250408 on /run/cephadm/e4069eba-694e-11ee-8db6-212e2dc638e7.lock 2023-10-12T22:31:27.074 INFO:teuthology.orchestra.run.smithi131.stderr:Lock 140242855250408 released on /run/cephadm/e4069eba-694e-11ee-8db6-212e2dc638e7.lock 2023-10-12T22:31:27.108 INFO:tasks.cephadm:Fetching config... 2023-10-12T22:31:27.108 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-10-12T22:31:27.109 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-10-12T22:31:27.127 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-10-12T22:31:27.127 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-10-12T22:31:27.128 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-10-12T22:31:27.185 INFO:tasks.cephadm:Fetching mon keyring... 2023-10-12T22:31:27.186 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-10-12T22:31:27.186 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/mon.smithi131/keyring of=/dev/stdout 2023-10-12T22:31:27.255 INFO:tasks.cephadm:Fetching pub ssh key... 2023-10-12T22:31:27.255 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-10-12T22:31:27.255 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-10-12T22:31:27.313 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-10-12T22:31:27.313 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlCssUDxr+d0RkbRblv9bheg4OhGeOyDhYM/fUdjLsOyCDPL61VVfpFoUG2eLOR8dEJOZiiU+3lfh16ELIs1b5Ex4by18zEjGNWYDtWjeBE2lF1ACqR2cJ9luZUJZWOXpf3xadXi3Npbmb0m4yq7MS/LuvoQtFr6pir3dJWAmG3er1g7oQsijdJLLlKIpe67DHiydNxo1Mpi4TDIGJ/MsvkJiRrUb03ZjnOsNDXvJ3pgD5+rrPY0+iBsP59ifYCMzmz/HDL7yrpndYfycPhAQKojZQ2drrdsfxU+vlAqa4BeeQ3y0xpN+D//jXuILLvUj2IFJ61rqOUzDvMKQAPswr/LqNNDk8jkz4l2Kh9OaTPyl90LA1KDYAvNHatV1bm0/vP++xgEcuH3mzpe6Wb4K/bzM3zcs3gm9RKdtTNT27nmv4OAZ5AcNtvGJxHI06a2Gmd+G7UCZvuQnXyWe94ZkBvf0BMZ7z6yPsCCnxkuFNQ1uBR+tg6RD3xPZkwuNS36M= ceph-e4069eba-694e-11ee-8db6-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-10-12T22:31:27.389 INFO:teuthology.orchestra.run.smithi131.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlCssUDxr+d0RkbRblv9bheg4OhGeOyDhYM/fUdjLsOyCDPL61VVfpFoUG2eLOR8dEJOZiiU+3lfh16ELIs1b5Ex4by18zEjGNWYDtWjeBE2lF1ACqR2cJ9luZUJZWOXpf3xadXi3Npbmb0m4yq7MS/LuvoQtFr6pir3dJWAmG3er1g7oQsijdJLLlKIpe67DHiydNxo1Mpi4TDIGJ/MsvkJiRrUb03ZjnOsNDXvJ3pgD5+rrPY0+iBsP59ifYCMzmz/HDL7yrpndYfycPhAQKojZQ2drrdsfxU+vlAqa4BeeQ3y0xpN+D//jXuILLvUj2IFJ61rqOUzDvMKQAPswr/LqNNDk8jkz4l2Kh9OaTPyl90LA1KDYAvNHatV1bm0/vP++xgEcuH3mzpe6Wb4K/bzM3zcs3gm9RKdtTNT27nmv4OAZ5AcNtvGJxHI06a2Gmd+G7UCZvuQnXyWe94ZkBvf0BMZ7z6yPsCCnxkuFNQ1uBR+tg6RD3xPZkwuNS36M= ceph-e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:31:27.400 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlCssUDxr+d0RkbRblv9bheg4OhGeOyDhYM/fUdjLsOyCDPL61VVfpFoUG2eLOR8dEJOZiiU+3lfh16ELIs1b5Ex4by18zEjGNWYDtWjeBE2lF1ACqR2cJ9luZUJZWOXpf3xadXi3Npbmb0m4yq7MS/LuvoQtFr6pir3dJWAmG3er1g7oQsijdJLLlKIpe67DHiydNxo1Mpi4TDIGJ/MsvkJiRrUb03ZjnOsNDXvJ3pgD5+rrPY0+iBsP59ifYCMzmz/HDL7yrpndYfycPhAQKojZQ2drrdsfxU+vlAqa4BeeQ3y0xpN+D//jXuILLvUj2IFJ61rqOUzDvMKQAPswr/LqNNDk8jkz4l2Kh9OaTPyl90LA1KDYAvNHatV1bm0/vP++xgEcuH3mzpe6Wb4K/bzM3zcs3gm9RKdtTNT27nmv4OAZ5AcNtvGJxHI06a2Gmd+G7UCZvuQnXyWe94ZkBvf0BMZ7z6yPsCCnxkuFNQ1uBR+tg6RD3xPZkwuNS36M= ceph-e4069eba-694e-11ee-8db6-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-10-12T22:31:27.446 INFO:teuthology.orchestra.run.smithi195.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlCssUDxr+d0RkbRblv9bheg4OhGeOyDhYM/fUdjLsOyCDPL61VVfpFoUG2eLOR8dEJOZiiU+3lfh16ELIs1b5Ex4by18zEjGNWYDtWjeBE2lF1ACqR2cJ9luZUJZWOXpf3xadXi3Npbmb0m4yq7MS/LuvoQtFr6pir3dJWAmG3er1g7oQsijdJLLlKIpe67DHiydNxo1Mpi4TDIGJ/MsvkJiRrUb03ZjnOsNDXvJ3pgD5+rrPY0+iBsP59ifYCMzmz/HDL7yrpndYfycPhAQKojZQ2drrdsfxU+vlAqa4BeeQ3y0xpN+D//jXuILLvUj2IFJ61rqOUzDvMKQAPswr/LqNNDk8jkz4l2Kh9OaTPyl90LA1KDYAvNHatV1bm0/vP++xgEcuH3mzpe6Wb4K/bzM3zcs3gm9RKdtTNT27nmv4OAZ5AcNtvGJxHI06a2Gmd+G7UCZvuQnXyWe94ZkBvf0BMZ7z6yPsCCnxkuFNQ1uBR+tg6RD3xPZkwuNS36M= ceph-e4069eba-694e-11ee-8db6-212e2dc638e7 2023-10-12T22:31:27.458 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-10-12T22:31:28.805 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.803+0000 7f7681073700 1 -- 172.21.15.131:0/278172518 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f767c060280 msgr2=0x7f767c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:28.806 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.803+0000 7f7681073700 1 --2- 172.21.15.131:0/278172518 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f767c060280 0x7f767c0f90f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f7670009fb0 tx=0x7f7670009b30).stop 2023-10-12T22:31:28.806 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.803+0000 7f7681073700 1 -- 172.21.15.131:0/278172518 shutdown_connections 2023-10-12T22:31:28.806 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.803+0000 7f7681073700 1 --2- 172.21.15.131:0/278172518 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f767c060280 0x7f767c0f90f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:28.806 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.803+0000 7f7681073700 1 -- 172.21.15.131:0/278172518 >> 172.21.15.131:0/278172518 conn(0x7f767c0f0b70 msgr2=0x7f767c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:28.806 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.803+0000 7f7681073700 1 -- 172.21.15.131:0/278172518 shutdown_connections 2023-10-12T22:31:28.807 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.803+0000 7f7681073700 1 -- 172.21.15.131:0/278172518 wait complete. 2023-10-12T22:31:28.807 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.804+0000 7f7681073700 1 Processor -- start 2023-10-12T22:31:28.807 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.804+0000 7f7681073700 1 -- start start 2023-10-12T22:31:28.807 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.804+0000 7f7681073700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f767c060280 0x7f767c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:28.807 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.804+0000 7f7681073700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0f4ff0 con 0x7f767c060280 2023-10-12T22:31:28.808 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.804+0000 7f767ad9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f767c060280 0x7f767c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:28.808 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.804+0000 7f767ad9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f767c060280 0x7f767c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:34388/0 (socket says 172.21.15.131:34388) 2023-10-12T22:31:28.808 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.804+0000 7f767ad9d700 1 -- 172.21.15.131:0/3388692767 learned_addr learned my addr 172.21.15.131:0/3388692767 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:28.809 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.805+0000 7f767ad9d700 1 -- 172.21.15.131:0/3388692767 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7670005040 con 0x7f767c060280 2023-10-12T22:31:28.810 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.805+0000 7f767ad9d700 1 --2- 172.21.15.131:0/3388692767 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f767c060280 0x7f767c0f6b70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f7670013510 tx=0x7f767000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:28.810 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.806+0000 7f76637fe700 1 -- 172.21.15.131:0/3388692767 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7670009320 con 0x7f767c060280 2023-10-12T22:31:28.810 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.806+0000 7f7681073700 1 -- 172.21.15.131:0/3388692767 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767c0f5190 con 0x7f767c060280 2023-10-12T22:31:28.810 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.806+0000 7f76637fe700 1 -- 172.21.15.131:0/3388692767 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7670013f00 con 0x7f767c060280 2023-10-12T22:31:28.811 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.806+0000 7f76637fe700 1 -- 172.21.15.131:0/3388692767 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7670014dc0 con 0x7f767c060280 2023-10-12T22:31:28.811 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.806+0000 7f7681073700 1 -- 172.21.15.131:0/3388692767 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767c0f5620 con 0x7f767c060280 2023-10-12T22:31:28.811 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.807+0000 7f76637fe700 1 -- 172.21.15.131:0/3388692767 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f767001d3d0 con 0x7f767c060280 2023-10-12T22:31:28.811 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.807+0000 7f7681073700 1 -- 172.21.15.131:0/3388692767 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767c059100 con 0x7f767c060280 2023-10-12T22:31:28.811 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.807+0000 7f76637fe700 1 --2- 172.21.15.131:0/3388692767 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f766402ff60 0x7f7664032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:28.812 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.807+0000 7f76637fe700 1 -- 172.21.15.131:0/3388692767 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76700095f0 con 0x7f767c060280 2023-10-12T22:31:28.812 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.807+0000 7f767a59c700 1 --2- 172.21.15.131:0/3388692767 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f766402ff60 0x7f7664032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:28.812 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.807+0000 7f767a59c700 1 --2- 172.21.15.131:0/3388692767 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f766402ff60 0x7f7664032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f766c0096a0 tx=0x7f766c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:28.813 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.811+0000 7f76637fe700 1 -- 172.21.15.131:0/3388692767 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7670012920 con 0x7f767c060280 2023-10-12T22:31:28.974 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.973+0000 7f7681073700 1 -- 172.21.15.131:0/3388692767 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f767c0fc0d0 con 0x7f767c060280 2023-10-12T22:31:28.977 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.976+0000 7f76637fe700 1 -- 172.21.15.131:0/3388692767 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7670012330 con 0x7f767c060280 2023-10-12T22:31:28.979 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.978+0000 7f7681073700 1 -- 172.21.15.131:0/3388692767 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f766402ff60 msgr2=0x7f7664032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:28.979 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.978+0000 7f7681073700 1 --2- 172.21.15.131:0/3388692767 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f766402ff60 0x7f7664032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f766c0096a0 tx=0x7f766c006b40).stop 2023-10-12T22:31:28.980 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.978+0000 7f7681073700 1 -- 172.21.15.131:0/3388692767 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f767c060280 msgr2=0x7f767c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:28.980 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.978+0000 7f7681073700 1 --2- 172.21.15.131:0/3388692767 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f767c060280 0x7f767c0f6b70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f7670013510 tx=0x7f767000b040).stop 2023-10-12T22:31:28.980 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.978+0000 7f7681073700 1 -- 172.21.15.131:0/3388692767 shutdown_connections 2023-10-12T22:31:28.980 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.978+0000 7f7681073700 1 --2- 172.21.15.131:0/3388692767 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f766402ff60 0x7f7664032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:28.980 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.978+0000 7f7681073700 1 --2- 172.21.15.131:0/3388692767 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f767c060280 0x7f767c0f6b70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:28.981 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.978+0000 7f7681073700 1 -- 172.21.15.131:0/3388692767 >> 172.21.15.131:0/3388692767 conn(0x7f767c0f0b70 msgr2=0x7f767c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:28.981 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.978+0000 7f7681073700 1 -- 172.21.15.131:0/3388692767 shutdown_connections 2023-10-12T22:31:28.981 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:28.978+0000 7f7681073700 1 -- 172.21.15.131:0/3388692767 wait complete. 2023-10-12T22:31:30.063 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:28.807140+0000 mon.smithi131 (mon.0) 112 : cluster [DBG] mgrmap e16: smithi131.efktma(active, since 6s) 2023-10-12T22:31:30.063 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:28.977323+0000 mon.smithi131 (mon.0) 113 : audit [INF] from='client.? 172.21.15.131:0/3388692767' entity='client.admin' 2023-10-12T22:31:30.081 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi195 2023-10-12T22:31:30.081 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-12T22:31:30.081 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.conf 2023-10-12T22:31:30.103 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2023-10-12T22:31:30.104 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-12T22:31:30.166 INFO:tasks.cephadm:Adding host smithi195 to orchestrator... 2023-10-12T22:31:30.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph orch host add smithi195 2023-10-12T22:31:31.685 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:31 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:30.603099+0000 mon.smithi131 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:31:31.685 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:31 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:30.879327+0000 mon.smithi131 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:31:32.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:31:30.885915+0000 mgr.smithi131.efktma (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi131 on smithi131 2023-10-12T22:31:44.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:43 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:42.797618+0000 mgr.smithi131.efktma (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:31:46.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:45 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:44.797878+0000 mgr.smithi131.efktma (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:31:48.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:47 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:46.798215+0000 mgr.smithi131.efktma (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:31:50.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:49 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:48.798559+0000 mgr.smithi131.efktma (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:31:52.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:51 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:50.798989+0000 mgr.smithi131.efktma (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:31:52.843 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.841+0000 7fafec82d700 1 -- 172.21.15.195:0/2226374799 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe80f6510 msgr2=0x7fafe80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:52.844 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.841+0000 7fafec82d700 1 --2- 172.21.15.195:0/2226374799 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe80f6510 0x7fafe80f6930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fafdc009fb0 tx=0x7fafdc009b30).stop 2023-10-12T22:31:52.844 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.842+0000 7fafec82d700 1 -- 172.21.15.195:0/2226374799 shutdown_connections 2023-10-12T22:31:52.844 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.842+0000 7fafec82d700 1 --2- 172.21.15.195:0/2226374799 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe80f6510 0x7fafe80f6930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:52.844 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.842+0000 7fafec82d700 1 -- 172.21.15.195:0/2226374799 >> 172.21.15.195:0/2226374799 conn(0x7fafe80f1c70 msgr2=0x7fafe80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:52.845 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.842+0000 7fafec82d700 1 -- 172.21.15.195:0/2226374799 shutdown_connections 2023-10-12T22:31:52.845 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.842+0000 7fafec82d700 1 -- 172.21.15.195:0/2226374799 wait complete. 2023-10-12T22:31:52.845 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.842+0000 7fafec82d700 1 Processor -- start 2023-10-12T22:31:52.845 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.842+0000 7fafec82d700 1 -- start start 2023-10-12T22:31:52.845 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.842+0000 7fafec82d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe80f6510 0x7fafe8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:52.846 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.842+0000 7fafec82d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe8106c20 con 0x7fafe80f6510 2023-10-12T22:31:52.846 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.843+0000 7fafe659c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe80f6510 0x7fafe8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:52.846 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.843+0000 7fafe659c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe80f6510 0x7fafe8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:41616/0 (socket says 172.21.15.195:41616) 2023-10-12T22:31:52.846 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.843+0000 7fafe659c700 1 -- 172.21.15.195:0/3751273472 learned_addr learned my addr 172.21.15.195:0/3751273472 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:31:52.846 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.843+0000 7fafe659c700 1 -- 172.21.15.195:0/3751273472 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc005040 con 0x7fafe80f6510 2023-10-12T22:31:52.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.843+0000 7fafe659c700 1 --2- 172.21.15.195:0/3751273472 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe80f6510 0x7fafe8106730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fafdc014000 tx=0x7fafdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:52.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.843+0000 7fafd6ffd700 1 -- 172.21.15.195:0/3751273472 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafdc0095f0 con 0x7fafe80f6510 2023-10-12T22:31:52.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.843+0000 7fafec82d700 1 -- 172.21.15.195:0/3751273472 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe8106e20 con 0x7fafe80f6510 2023-10-12T22:31:52.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.843+0000 7fafd6ffd700 1 -- 172.21.15.195:0/3751273472 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafdc007e70 con 0x7fafe80f6510 2023-10-12T22:31:52.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.843+0000 7fafd6ffd700 1 -- 172.21.15.195:0/3751273472 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafdc014ea0 con 0x7fafe80f6510 2023-10-12T22:31:52.848 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.843+0000 7fafec82d700 1 -- 172.21.15.195:0/3751273472 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe81072b0 con 0x7fafe80f6510 2023-10-12T22:31:52.849 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.844+0000 7fafd6ffd700 1 -- 172.21.15.195:0/3751273472 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fafdc025070 con 0x7fafe80f6510 2023-10-12T22:31:52.849 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.844+0000 7fafec82d700 1 -- 172.21.15.195:0/3751273472 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe8045ad0 con 0x7fafe80f6510 2023-10-12T22:31:52.849 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.844+0000 7fafd6ffd700 1 --2- 172.21.15.195:0/3751273472 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fafd002ff60 0x7fafd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:52.849 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.844+0000 7fafd6ffd700 1 -- 172.21.15.195:0/3751273472 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fafdc0409a0 con 0x7fafe80f6510 2023-10-12T22:31:52.849 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.844+0000 7fafe5d9b700 1 --2- 172.21.15.195:0/3751273472 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fafd002ff60 0x7fafd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:52.850 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.845+0000 7fafe5d9b700 1 --2- 172.21.15.195:0/3751273472 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fafd002ff60 0x7fafd0032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fafd80096a0 tx=0x7fafd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:52.850 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:52.849+0000 7fafd6ffd700 1 -- 172.21.15.195:0/3751273472 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafdc00f4e0 con 0x7fafe80f6510 2023-10-12T22:31:53.019 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:53.019+0000 7fafec82d700 1 -- 172.21.15.195:0/3751273472 --> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}) v1 -- 0x7fafe80f2e70 con 0x7fafd002ff60 2023-10-12T22:31:53.612 INFO:teuthology.orchestra.run.smithi195.stdout:Added host 'smithi195' 2023-10-12T22:31:53.612 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:53.611+0000 7fafd6ffd700 1 -- 172.21.15.195:0/3751273472 <== mgr.14162 v2:172.21.15.131:6800/3705786429 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fafe80f2e70 con 0x7fafd002ff60 2023-10-12T22:31:53.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:53.613+0000 7fafec82d700 1 -- 172.21.15.195:0/3751273472 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fafd002ff60 msgr2=0x7fafd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:53.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:53.613+0000 7fafec82d700 1 --2- 172.21.15.195:0/3751273472 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fafd002ff60 0x7fafd0032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fafd80096a0 tx=0x7fafd8006b40).stop 2023-10-12T22:31:53.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:53.613+0000 7fafec82d700 1 -- 172.21.15.195:0/3751273472 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe80f6510 msgr2=0x7fafe8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:53.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:53.613+0000 7fafec82d700 1 --2- 172.21.15.195:0/3751273472 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe80f6510 0x7fafe8106730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fafdc014000 tx=0x7fafdc00b040).stop 2023-10-12T22:31:53.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:53.613+0000 7fafec82d700 1 -- 172.21.15.195:0/3751273472 shutdown_connections 2023-10-12T22:31:53.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:53.613+0000 7fafec82d700 1 --2- 172.21.15.195:0/3751273472 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fafd002ff60 0x7fafd0032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:53.615 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:53.613+0000 7fafec82d700 1 --2- 172.21.15.195:0/3751273472 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe80f6510 0x7fafe8106730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:53.615 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:53.613+0000 7fafec82d700 1 -- 172.21.15.195:0/3751273472 >> 172.21.15.195:0/3751273472 conn(0x7fafe80f1c70 msgr2=0x7fafe80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:53.615 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:53.613+0000 7fafec82d700 1 -- 172.21.15.195:0/3751273472 shutdown_connections 2023-10-12T22:31:53.615 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:53.613+0000 7fafec82d700 1 -- 172.21.15.195:0/3751273472 wait complete. 2023-10-12T22:31:53.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph orch host ls --format=json 2023-10-12T22:31:54.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:52.799277+0000 mgr.smithi131.efktma (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:31:54.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:53.020089+0000 mgr.smithi131.efktma (mgr.14162) 12 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:31:54.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:53.610344+0000 mon.smithi131 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:31:54.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:31:53.610772+0000 mgr.smithi131.efktma (mgr.14162) 13 : cephadm [INF] Added host smithi195 2023-10-12T22:31:55.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.044+0000 7f8c5f24e700 1 -- 172.21.15.195:0/2677880792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c58060220 msgr2=0x7f8c580f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:55.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.044+0000 7f8c5f24e700 1 --2- 172.21.15.195:0/2677880792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c58060220 0x7f8c580f9150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f8c4c009fb0 tx=0x7f8c4c009b30).stop 2023-10-12T22:31:55.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.044+0000 7f8c5f24e700 1 -- 172.21.15.195:0/2677880792 shutdown_connections 2023-10-12T22:31:55.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.044+0000 7f8c5f24e700 1 --2- 172.21.15.195:0/2677880792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c58060220 0x7f8c580f9150 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:55.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.044+0000 7f8c5f24e700 1 -- 172.21.15.195:0/2677880792 >> 172.21.15.195:0/2677880792 conn(0x7f8c580f0b70 msgr2=0x7f8c580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:55.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.044+0000 7f8c5f24e700 1 -- 172.21.15.195:0/2677880792 shutdown_connections 2023-10-12T22:31:55.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.044+0000 7f8c5f24e700 1 -- 172.21.15.195:0/2677880792 wait complete. 2023-10-12T22:31:55.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.045+0000 7f8c5f24e700 1 Processor -- start 2023-10-12T22:31:55.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.045+0000 7f8c5f24e700 1 -- start start 2023-10-12T22:31:55.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.045+0000 7f8c5f24e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c58060220 0x7f8c581066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:55.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.045+0000 7f8c5f24e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c58106bd0 con 0x7f8c58060220 2023-10-12T22:31:55.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.045+0000 7f8c5cfea700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c58060220 0x7f8c581066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:55.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.045+0000 7f8c5cfea700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c58060220 0x7f8c581066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:41620/0 (socket says 172.21.15.195:41620) 2023-10-12T22:31:55.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.045+0000 7f8c5cfea700 1 -- 172.21.15.195:0/226385213 learned_addr learned my addr 172.21.15.195:0/226385213 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:31:55.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.046+0000 7f8c5cfea700 1 -- 172.21.15.195:0/226385213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c005040 con 0x7f8c58060220 2023-10-12T22:31:55.050 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.046+0000 7f8c5cfea700 1 --2- 172.21.15.195:0/226385213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c58060220 0x7f8c581066e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f8c4c013510 tx=0x7f8c4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:55.050 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.046+0000 7f8c557fa700 1 -- 172.21.15.195:0/226385213 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c4c009320 con 0x7f8c58060220 2023-10-12T22:31:55.050 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.046+0000 7f8c5f24e700 1 -- 172.21.15.195:0/226385213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c58106dd0 con 0x7f8c58060220 2023-10-12T22:31:55.050 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.046+0000 7f8c557fa700 1 -- 172.21.15.195:0/226385213 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c4c013f00 con 0x7f8c58060220 2023-10-12T22:31:55.050 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.046+0000 7f8c557fa700 1 -- 172.21.15.195:0/226385213 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c4c014dc0 con 0x7f8c58060220 2023-10-12T22:31:55.051 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.046+0000 7f8c5f24e700 1 -- 172.21.15.195:0/226385213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c58107260 con 0x7f8c58060220 2023-10-12T22:31:55.052 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.047+0000 7f8c557fa700 1 -- 172.21.15.195:0/226385213 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c4c014420 con 0x7f8c58060220 2023-10-12T22:31:55.052 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.047+0000 7f8c5f24e700 1 -- 172.21.15.195:0/226385213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c58045ad0 con 0x7f8c58060220 2023-10-12T22:31:55.052 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.047+0000 7f8c557fa700 1 --2- 172.21.15.195:0/226385213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8c4802ff60 0x7f8c48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:55.052 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.047+0000 7f8c557fa700 1 -- 172.21.15.195:0/226385213 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c4c0410f0 con 0x7f8c58060220 2023-10-12T22:31:55.052 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.047+0000 7f8c57fff700 1 --2- 172.21.15.195:0/226385213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8c4802ff60 0x7f8c48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:55.053 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.048+0000 7f8c57fff700 1 --2- 172.21.15.195:0/226385213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8c4802ff60 0x7f8c48032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8c440096a0 tx=0x7f8c44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:55.053 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.052+0000 7f8c557fa700 1 -- 172.21.15.195:0/226385213 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c4c0095f0 con 0x7f8c58060220 2023-10-12T22:31:55.225 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.224+0000 7f8c5f24e700 1 -- 172.21.15.195:0/226385213 --> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c580f1d70 con 0x7f8c4802ff60 2023-10-12T22:31:55.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.226+0000 7f8c557fa700 1 -- 172.21.15.195:0/226385213 <== mgr.14162 v2:172.21.15.131:6800/3705786429 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f8c580f1d70 con 0x7f8c4802ff60 2023-10-12T22:31:55.227 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:31:55.227 INFO:teuthology.orchestra.run.smithi195.stdout:[{"addr": "172.21.15.131", "hostname": "smithi131", "labels": [], "status": ""}, {"addr": "smithi195", "hostname": "smithi195", "labels": [], "status": ""}] 2023-10-12T22:31:55.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.228+0000 7f8c5f24e700 1 -- 172.21.15.195:0/226385213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8c4802ff60 msgr2=0x7f8c48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:55.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.228+0000 7f8c5f24e700 1 --2- 172.21.15.195:0/226385213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8c4802ff60 0x7f8c48032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f8c440096a0 tx=0x7f8c44006b40).stop 2023-10-12T22:31:55.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.228+0000 7f8c5f24e700 1 -- 172.21.15.195:0/226385213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c58060220 msgr2=0x7f8c581066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:55.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.228+0000 7f8c5f24e700 1 --2- 172.21.15.195:0/226385213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c58060220 0x7f8c581066e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f8c4c013510 tx=0x7f8c4c00b040).stop 2023-10-12T22:31:55.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.228+0000 7f8c5f24e700 1 -- 172.21.15.195:0/226385213 shutdown_connections 2023-10-12T22:31:55.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.228+0000 7f8c5f24e700 1 --2- 172.21.15.195:0/226385213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8c4802ff60 0x7f8c48032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:55.231 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.228+0000 7f8c5f24e700 1 --2- 172.21.15.195:0/226385213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8c58060220 0x7f8c581066e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:55.231 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.228+0000 7f8c5f24e700 1 -- 172.21.15.195:0/226385213 >> 172.21.15.195:0/226385213 conn(0x7f8c580f0b70 msgr2=0x7f8c580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:55.231 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.228+0000 7f8c5f24e700 1 -- 172.21.15.195:0/226385213 shutdown_connections 2023-10-12T22:31:55.232 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:55.228+0000 7f8c5f24e700 1 -- 172.21.15.195:0/226385213 wait complete. 2023-10-12T22:31:55.594 INFO:tasks.cephadm:Setting crush tunables to default 2023-10-12T22:31:55.594 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph osd crush tunables default 2023-10-12T22:31:56.692 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.690+0000 7fe4b4d87700 1 -- 172.21.15.131:0/1661111358 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4b00f6510 msgr2=0x7fe4b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:56.693 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.690+0000 7fe4b4d87700 1 --2- 172.21.15.131:0/1661111358 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4b00f6510 0x7fe4b00f6930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fe4a0009fb0 tx=0x7fe4a0009b30).stop 2023-10-12T22:31:56.693 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.691+0000 7fe4b4d87700 1 -- 172.21.15.131:0/1661111358 shutdown_connections 2023-10-12T22:31:56.693 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.691+0000 7fe4b4d87700 1 --2- 172.21.15.131:0/1661111358 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4b00f6510 0x7fe4b00f6930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:56.693 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.691+0000 7fe4b4d87700 1 -- 172.21.15.131:0/1661111358 >> 172.21.15.131:0/1661111358 conn(0x7fe4b00f1c70 msgr2=0x7fe4b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:56.694 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.691+0000 7fe4b4d87700 1 -- 172.21.15.131:0/1661111358 shutdown_connections 2023-10-12T22:31:56.694 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.691+0000 7fe4b4d87700 1 -- 172.21.15.131:0/1661111358 wait complete. 2023-10-12T22:31:56.694 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.691+0000 7fe4b4d87700 1 Processor -- start 2023-10-12T22:31:56.694 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.691+0000 7fe4b4d87700 1 -- start start 2023-10-12T22:31:56.694 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.691+0000 7fe4b4d87700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4b00f6510 0x7fe4b006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:56.695 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.691+0000 7fe4b4d87700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b0069e20 con 0x7fe4b00f6510 2023-10-12T22:31:56.695 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.691+0000 7fe4ae59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4b00f6510 0x7fe4b006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:56.695 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.691+0000 7fe4ae59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4b00f6510 0x7fe4b006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:35030/0 (socket says 172.21.15.131:35030) 2023-10-12T22:31:56.695 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.691+0000 7fe4ae59c700 1 -- 172.21.15.131:0/2231307232 learned_addr learned my addr 172.21.15.131:0/2231307232 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-10-12T22:31:56.695 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.692+0000 7fe4ae59c700 1 -- 172.21.15.131:0/2231307232 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a0005040 con 0x7fe4b00f6510 2023-10-12T22:31:56.696 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.692+0000 7fe4ae59c700 1 --2- 172.21.15.131:0/2231307232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4b00f6510 0x7fe4b006b9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe4a0013510 tx=0x7fe4a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:56.696 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.692+0000 7fe49effd700 1 -- 172.21.15.131:0/2231307232 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4a0009320 con 0x7fe4b00f6510 2023-10-12T22:31:56.696 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.692+0000 7fe49effd700 1 -- 172.21.15.131:0/2231307232 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4a0013f00 con 0x7fe4b00f6510 2023-10-12T22:31:56.696 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.692+0000 7fe4b4d87700 1 -- 172.21.15.131:0/2231307232 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b0069fc0 con 0x7fe4b00f6510 2023-10-12T22:31:56.696 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.692+0000 7fe49effd700 1 -- 172.21.15.131:0/2231307232 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4a0014d90 con 0x7fe4b00f6510 2023-10-12T22:31:56.696 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.692+0000 7fe4b4d87700 1 -- 172.21.15.131:0/2231307232 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b006a450 con 0x7fe4b00f6510 2023-10-12T22:31:56.697 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.693+0000 7fe49effd700 1 -- 172.21.15.131:0/2231307232 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4a00263a0 con 0x7fe4b00f6510 2023-10-12T22:31:56.698 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.693+0000 7fe49effd700 1 --2- 172.21.15.131:0/2231307232 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe49802ffb0 0x7fe498032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:56.698 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.693+0000 7fe49effd700 1 -- 172.21.15.131:0/2231307232 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4a0040fa0 con 0x7fe4b00f6510 2023-10-12T22:31:56.698 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.693+0000 7fe4b4d87700 1 -- 172.21.15.131:0/2231307232 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b0059100 con 0x7fe4b00f6510 2023-10-12T22:31:56.698 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.693+0000 7fe4add9b700 1 --2- 172.21.15.131:0/2231307232 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe49802ffb0 0x7fe498032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:56.699 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.693+0000 7fe4add9b700 1 --2- 172.21.15.131:0/2231307232 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe49802ffb0 0x7fe498032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe4a40096a0 tx=0x7fe4a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:56.699 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.697+0000 7fe49effd700 1 -- 172.21.15.131:0/2231307232 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4a0012920 con 0x7fe4b00f6510 2023-10-12T22:31:56.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:54.799553+0000 mgr.smithi131.efktma (mgr.14162) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:31:56.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:55.225939+0000 mgr.smithi131.efktma (mgr.14162) 15 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-12T22:31:56.851 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:56.851+0000 7fe4b4d87700 1 -- 172.21.15.131:0/2231307232 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe4b006ab50 con 0x7fe4b00f6510 2023-10-12T22:31:57.620 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:57.620+0000 7fe49effd700 1 -- 172.21.15.131:0/2231307232 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe4a002b030 con 0x7fe4b00f6510 2023-10-12T22:31:57.622 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:57.622+0000 7fe4b4d87700 1 -- 172.21.15.131:0/2231307232 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe49802ffb0 msgr2=0x7fe498032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:57.622 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:57.622+0000 7fe4b4d87700 1 --2- 172.21.15.131:0/2231307232 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe49802ffb0 0x7fe498032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe4a40096a0 tx=0x7fe4a4006b40).stop 2023-10-12T22:31:57.622 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:57.622+0000 7fe4b4d87700 1 -- 172.21.15.131:0/2231307232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4b00f6510 msgr2=0x7fe4b006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:57.623 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:57.622+0000 7fe4b4d87700 1 --2- 172.21.15.131:0/2231307232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4b00f6510 0x7fe4b006b9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe4a0013510 tx=0x7fe4a000b040).stop 2023-10-12T22:31:57.623 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:57.622+0000 7fe4b4d87700 1 -- 172.21.15.131:0/2231307232 shutdown_connections 2023-10-12T22:31:57.623 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:57.622+0000 7fe4b4d87700 1 --2- 172.21.15.131:0/2231307232 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe49802ffb0 0x7fe498032470 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:57.623 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:57.622+0000 7fe4b4d87700 1 --2- 172.21.15.131:0/2231307232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4b00f6510 0x7fe4b006b9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:57.623 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:57.622+0000 7fe4b4d87700 1 -- 172.21.15.131:0/2231307232 >> 172.21.15.131:0/2231307232 conn(0x7fe4b00f1c70 msgr2=0x7fe4b00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:57.623 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:57.622+0000 7fe4b4d87700 1 -- 172.21.15.131:0/2231307232 shutdown_connections 2023-10-12T22:31:57.624 INFO:teuthology.orchestra.run.smithi131.stderr:2023-10-12T22:31:57.622+0000 7fe4b4d87700 1 -- 172.21.15.131:0/2231307232 wait complete. 2023-10-12T22:31:57.624 INFO:teuthology.orchestra.run.smithi131.stderr:adjusted tunables profile to default 2023-10-12T22:31:57.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:57 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:56.852247+0000 mon.smithi131 (mon.0) 117 : audit [INF] from='client.? 172.21.15.131:0/2231307232' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-10-12T22:31:57.976 INFO:tasks.cephadm:Adding mon.smithi131 on smithi131 2023-10-12T22:31:57.976 INFO:tasks.cephadm:Adding mon.smithi195 on smithi195 2023-10-12T22:31:57.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph orch apply mon '2;smithi131:172.21.15.131=smithi131;smithi195:172.21.15.195=smithi195' 2023-10-12T22:31:58.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:58 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:56.799826+0000 mgr.smithi131.efktma (mgr.14162) 16 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:31:58.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:58 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:57.619996+0000 mon.smithi131 (mon.0) 118 : audit [INF] from='client.? 172.21.15.131:0/2231307232' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-10-12T22:31:58.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:31:58 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:57.620048+0000 mon.smithi131 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-10-12T22:31:59.093 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.090+0000 7f3513fff700 1 -- 172.21.15.195:0/3794323868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35140f51a0 msgr2=0x7f35140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:59.093 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.091+0000 7f3513fff700 1 --2- 172.21.15.195:0/3794323868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35140f51a0 0x7f35140f55c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f3504009fb0 tx=0x7f3504009b30).stop 2023-10-12T22:31:59.094 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.091+0000 7f3513fff700 1 -- 172.21.15.195:0/3794323868 shutdown_connections 2023-10-12T22:31:59.094 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.091+0000 7f3513fff700 1 --2- 172.21.15.195:0/3794323868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35140f51a0 0x7f35140f55c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:59.094 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.091+0000 7f3513fff700 1 -- 172.21.15.195:0/3794323868 >> 172.21.15.195:0/3794323868 conn(0x7f35140f09a0 msgr2=0x7f35140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:59.094 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.091+0000 7f3513fff700 1 -- 172.21.15.195:0/3794323868 shutdown_connections 2023-10-12T22:31:59.094 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.091+0000 7f3513fff700 1 -- 172.21.15.195:0/3794323868 wait complete. 2023-10-12T22:31:59.095 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.091+0000 7f3513fff700 1 Processor -- start 2023-10-12T22:31:59.095 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.091+0000 7f3513fff700 1 -- start start 2023-10-12T22:31:59.095 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.091+0000 7f3513fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35140f51a0 0x7f35141064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:59.095 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.091+0000 7f3513fff700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35141069c0 con 0x7f35140f51a0 2023-10-12T22:31:59.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.092+0000 7f3512ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35140f51a0 0x7f35141064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:59.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.092+0000 7f3512ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35140f51a0 0x7f35141064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:53756/0 (socket says 172.21.15.195:53756) 2023-10-12T22:31:59.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.092+0000 7f3512ffd700 1 -- 172.21.15.195:0/3179964122 learned_addr learned my addr 172.21.15.195:0/3179964122 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:31:59.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.092+0000 7f3512ffd700 1 -- 172.21.15.195:0/3179964122 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3504005040 con 0x7f35140f51a0 2023-10-12T22:31:59.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.092+0000 7f3512ffd700 1 --2- 172.21.15.195:0/3179964122 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35140f51a0 0x7f35141064d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f3504013510 tx=0x7f350400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:59.098 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.092+0000 7f34fb7fe700 1 -- 172.21.15.195:0/3179964122 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3504009320 con 0x7f35140f51a0 2023-10-12T22:31:59.098 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.092+0000 7f3513fff700 1 -- 172.21.15.195:0/3179964122 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3514106bc0 con 0x7f35140f51a0 2023-10-12T22:31:59.098 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.092+0000 7f34fb7fe700 1 -- 172.21.15.195:0/3179964122 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3504013f00 con 0x7f35140f51a0 2023-10-12T22:31:59.098 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.092+0000 7f34fb7fe700 1 -- 172.21.15.195:0/3179964122 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3504014dc0 con 0x7f35140f51a0 2023-10-12T22:31:59.098 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.092+0000 7f3513fff700 1 -- 172.21.15.195:0/3179964122 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3514107050 con 0x7f35140f51a0 2023-10-12T22:31:59.099 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.093+0000 7f34fb7fe700 1 -- 172.21.15.195:0/3179964122 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35040095f0 con 0x7f35140f51a0 2023-10-12T22:31:59.099 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.093+0000 7f3513fff700 1 -- 172.21.15.195:0/3179964122 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3514045ad0 con 0x7f35140f51a0 2023-10-12T22:31:59.099 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.094+0000 7f34fb7fe700 1 --2- 172.21.15.195:0/3179964122 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f34fc02ff60 0x7f34fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:31:59.099 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.094+0000 7f34fb7fe700 1 -- 172.21.15.195:0/3179964122 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3504041100 con 0x7f35140f51a0 2023-10-12T22:31:59.099 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.094+0000 7f35127fc700 1 --2- 172.21.15.195:0/3179964122 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f34fc02ff60 0x7f34fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:31:59.100 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.094+0000 7f35127fc700 1 --2- 172.21.15.195:0/3179964122 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f34fc02ff60 0x7f34fc032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f35080096a0 tx=0x7f3508006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:31:59.100 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.098+0000 7f34fb7fe700 1 -- 172.21.15.195:0/3179964122 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3504014420 con 0x7f35140f51a0 2023-10-12T22:31:59.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.275+0000 7f3513fff700 1 -- 172.21.15.195:0/3179964122 --> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi131:172.21.15.131=smithi131;smithi195:172.21.15.195=smithi195", "target": ["mon-mgr", ""]}) v1 -- 0x7f35140f28a0 con 0x7f34fc02ff60 2023-10-12T22:31:59.281 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.280+0000 7f34fb7fe700 1 -- 172.21.15.195:0/3179964122 <== mgr.14162 v2:172.21.15.131:6800/3705786429 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f35140f28a0 con 0x7f34fc02ff60 2023-10-12T22:31:59.282 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mon update... 2023-10-12T22:31:59.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.282+0000 7f3513fff700 1 -- 172.21.15.195:0/3179964122 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f34fc02ff60 msgr2=0x7f34fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:59.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.282+0000 7f3513fff700 1 --2- 172.21.15.195:0/3179964122 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f34fc02ff60 0x7f34fc032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f35080096a0 tx=0x7f3508006b40).stop 2023-10-12T22:31:59.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.282+0000 7f3513fff700 1 -- 172.21.15.195:0/3179964122 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35140f51a0 msgr2=0x7f35141064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:31:59.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.282+0000 7f3513fff700 1 --2- 172.21.15.195:0/3179964122 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35140f51a0 0x7f35141064d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f3504013510 tx=0x7f350400b040).stop 2023-10-12T22:31:59.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.282+0000 7f3513fff700 1 -- 172.21.15.195:0/3179964122 shutdown_connections 2023-10-12T22:31:59.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.282+0000 7f3513fff700 1 --2- 172.21.15.195:0/3179964122 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f34fc02ff60 0x7f34fc032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:59.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.282+0000 7f3513fff700 1 --2- 172.21.15.195:0/3179964122 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35140f51a0 0x7f35141064d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:31:59.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.282+0000 7f3513fff700 1 -- 172.21.15.195:0/3179964122 >> 172.21.15.195:0/3179964122 conn(0x7f35140f09a0 msgr2=0x7f351405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:31:59.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.282+0000 7f3513fff700 1 -- 172.21.15.195:0/3179964122 shutdown_connections 2023-10-12T22:31:59.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:31:59.282+0000 7f3513fff700 1 -- 172.21.15.195:0/3179964122 wait complete. 2023-10-12T22:31:59.646 DEBUG:teuthology.orchestra.run.smithi195:mon.smithi195> sudo journalctl -f -n 0 -u ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi195.service 2023-10-12T22:31:59.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:31:59.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:31:59.678 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:-- Logs begin at Thu 2023-10-12 22:18:09 UTC. -- 2023-10-12T22:32:00.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:31:58.800179+0000 mgr.smithi131.efktma (mgr.14162) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:00.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:59.277097+0000 mgr.smithi131.efktma (mgr.14162) 18 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi131:172.21.15.131=smithi131;smithi195:172.21.15.195=smithi195", "target": ["mon-mgr", ""]}]: dispatch 2023-10-12T22:32:00.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:31:59.278512+0000 mgr.smithi131.efktma (mgr.14162) 19 : cephadm [INF] Saving service mon spec with placement smithi131:172.21.15.131=smithi131;smithi195:172.21.15.195=smithi195;count:2 2023-10-12T22:32:00.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:31:59.281225+0000 mon.smithi131 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:32:00.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.764+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/2670043351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8bc00f6510 msgr2=0x7f8bc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:00.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.764+0000 7f8bc7f5f700 1 --2- 172.21.15.195:0/2670043351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8bc00f6510 0x7f8bc00f6930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f8bb0009fb0 tx=0x7f8bb0009b30).stop 2023-10-12T22:32:00.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.764+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/2670043351 shutdown_connections 2023-10-12T22:32:00.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.764+0000 7f8bc7f5f700 1 --2- 172.21.15.195:0/2670043351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8bc00f6510 0x7f8bc00f6930 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:00.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.764+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/2670043351 >> 172.21.15.195:0/2670043351 conn(0x7f8bc00f1c70 msgr2=0x7f8bc00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:00.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.764+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/2670043351 shutdown_connections 2023-10-12T22:32:00.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.764+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/2670043351 wait complete. 2023-10-12T22:32:00.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.764+0000 7f8bc7f5f700 1 Processor -- start 2023-10-12T22:32:00.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.764+0000 7f8bc7f5f700 1 -- start start 2023-10-12T22:32:00.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.765+0000 7f8bc7f5f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8bc00f6510 0x7f8bc0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:00.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.765+0000 7f8bc7f5f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc0106c20 con 0x7f8bc00f6510 2023-10-12T22:32:00.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.765+0000 7f8bc5cfb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8bc00f6510 0x7f8bc0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:00.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.765+0000 7f8bc5cfb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8bc00f6510 0x7f8bc0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:53762/0 (socket says 172.21.15.195:53762) 2023-10-12T22:32:00.770 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.765+0000 7f8bc5cfb700 1 -- 172.21.15.195:0/1614761521 learned_addr learned my addr 172.21.15.195:0/1614761521 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:00.770 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.765+0000 7f8bc5cfb700 1 -- 172.21.15.195:0/1614761521 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb0005040 con 0x7f8bc00f6510 2023-10-12T22:32:00.770 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.765+0000 7f8bc5cfb700 1 --2- 172.21.15.195:0/1614761521 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8bc00f6510 0x7f8bc0106730 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f8bb0014000 tx=0x7f8bb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:00.771 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.765+0000 7f8bb67fc700 1 -- 172.21.15.195:0/1614761521 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bb00095f0 con 0x7f8bc00f6510 2023-10-12T22:32:00.771 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.765+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/1614761521 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc0106e20 con 0x7f8bc00f6510 2023-10-12T22:32:00.771 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.766+0000 7f8bb67fc700 1 -- 172.21.15.195:0/1614761521 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8bb0007e70 con 0x7f8bc00f6510 2023-10-12T22:32:00.771 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.766+0000 7f8bb67fc700 1 -- 172.21.15.195:0/1614761521 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bb0014ea0 con 0x7f8bc00f6510 2023-10-12T22:32:00.771 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.766+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/1614761521 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc01072b0 con 0x7f8bc00f6510 2023-10-12T22:32:00.772 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.766+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/1614761521 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc0045ad0 con 0x7f8bc00f6510 2023-10-12T22:32:00.773 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.766+0000 7f8bb67fc700 1 -- 172.21.15.195:0/1614761521 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8bb0025070 con 0x7f8bc00f6510 2023-10-12T22:32:00.773 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.766+0000 7f8bb67fc700 1 --2- 172.21.15.195:0/1614761521 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8bac02ff60 0x7f8bac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:00.773 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.766+0000 7f8bb67fc700 1 -- 172.21.15.195:0/1614761521 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8bb00409f0 con 0x7f8bc00f6510 2023-10-12T22:32:00.773 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.767+0000 7f8bc54fa700 1 --2- 172.21.15.195:0/1614761521 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8bac02ff60 0x7f8bac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:00.773 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.767+0000 7f8bc54fa700 1 --2- 172.21.15.195:0/1614761521 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8bac02ff60 0x7f8bac032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f8bbc0096a0 tx=0x7f8bbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:00.774 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.771+0000 7f8bb67fc700 1 -- 172.21.15.195:0/1614761521 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8bb00079c0 con 0x7f8bc00f6510 2023-10-12T22:32:00.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.973+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/1614761521 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8bc00fbf60 con 0x7f8bc00f6510 2023-10-12T22:32:00.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.974+0000 7f8bb67fc700 1 -- 172.21.15.195:0/1614761521 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8bb000f300 con 0x7f8bc00f6510 2023-10-12T22:32:00.976 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:00.976 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:00.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.976+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/1614761521 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8bac02ff60 msgr2=0x7f8bac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:00.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.976+0000 7f8bc7f5f700 1 --2- 172.21.15.195:0/1614761521 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8bac02ff60 0x7f8bac032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f8bbc0096a0 tx=0x7f8bbc006b40).stop 2023-10-12T22:32:00.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.976+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/1614761521 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8bc00f6510 msgr2=0x7f8bc0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:00.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.976+0000 7f8bc7f5f700 1 --2- 172.21.15.195:0/1614761521 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8bc00f6510 0x7f8bc0106730 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f8bb0014000 tx=0x7f8bb000b040).stop 2023-10-12T22:32:00.978 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.976+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/1614761521 shutdown_connections 2023-10-12T22:32:00.978 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.976+0000 7f8bc7f5f700 1 --2- 172.21.15.195:0/1614761521 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8bac02ff60 0x7f8bac032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:00.978 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.976+0000 7f8bc7f5f700 1 --2- 172.21.15.195:0/1614761521 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8bc00f6510 0x7f8bc0106730 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:00.979 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.976+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/1614761521 >> 172.21.15.195:0/1614761521 conn(0x7f8bc00f1c70 msgr2=0x7f8bc00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:00.979 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.976+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/1614761521 shutdown_connections 2023-10-12T22:32:00.979 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:00.976+0000 7f8bc7f5f700 1 -- 172.21.15.195:0/1614761521 wait complete. 2023-10-12T22:32:00.979 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:01.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:01 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:00.975508+0000 mon.smithi131 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.195:0/1614761521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:02.402 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:02.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:02.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:02 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:00.800478+0000 mgr.smithi131.efktma (mgr.14162) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:03.515 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.509+0000 7f6947464700 1 -- 172.21.15.195:0/1509014089 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69400f5410 msgr2=0x7f69400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:03.515 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.509+0000 7f6947464700 1 --2- 172.21.15.195:0/1509014089 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69400f5410 0x7f69400f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f693c009fb0 tx=0x7f693c009b30).stop 2023-10-12T22:32:03.516 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.509+0000 7f6947464700 1 -- 172.21.15.195:0/1509014089 shutdown_connections 2023-10-12T22:32:03.516 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.509+0000 7f6947464700 1 --2- 172.21.15.195:0/1509014089 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69400f5410 0x7f69400f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:03.516 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.509+0000 7f6947464700 1 -- 172.21.15.195:0/1509014089 >> 172.21.15.195:0/1509014089 conn(0x7f69400f0b70 msgr2=0x7f69400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:03.516 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.510+0000 7f6947464700 1 -- 172.21.15.195:0/1509014089 shutdown_connections 2023-10-12T22:32:03.517 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.510+0000 7f6947464700 1 -- 172.21.15.195:0/1509014089 wait complete. 2023-10-12T22:32:03.517 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.510+0000 7f6947464700 1 Processor -- start 2023-10-12T22:32:03.517 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.510+0000 7f6947464700 1 -- start start 2023-10-12T22:32:03.518 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.510+0000 7f6947464700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69400f5410 0x7f6940105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:03.518 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.510+0000 7f6947464700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6940105b20 con 0x7f69400f5410 2023-10-12T22:32:03.518 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.511+0000 7f6945200700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69400f5410 0x7f6940105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:03.519 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.511+0000 7f6945200700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69400f5410 0x7f6940105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:53780/0 (socket says 172.21.15.195:53780) 2023-10-12T22:32:03.519 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.511+0000 7f6945200700 1 -- 172.21.15.195:0/3492779703 learned_addr learned my addr 172.21.15.195:0/3492779703 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:03.519 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.511+0000 7f6945200700 1 -- 172.21.15.195:0/3492779703 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693c005040 con 0x7f69400f5410 2023-10-12T22:32:03.519 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.511+0000 7f6945200700 1 --2- 172.21.15.195:0/3492779703 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69400f5410 0x7f6940105630 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f693c013510 tx=0x7f693c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:03.519 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.511+0000 7f6935ffb700 1 -- 172.21.15.195:0/3492779703 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f693c009320 con 0x7f69400f5410 2023-10-12T22:32:03.520 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.511+0000 7f6947464700 1 -- 172.21.15.195:0/3492779703 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6940105d20 con 0x7f69400f5410 2023-10-12T22:32:03.520 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.511+0000 7f6935ffb700 1 -- 172.21.15.195:0/3492779703 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f693c013f00 con 0x7f69400f5410 2023-10-12T22:32:03.520 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.511+0000 7f6935ffb700 1 -- 172.21.15.195:0/3492779703 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f693c014d90 con 0x7f69400f5410 2023-10-12T22:32:03.520 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.511+0000 7f6947464700 1 -- 172.21.15.195:0/3492779703 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69401061b0 con 0x7f69400f5410 2023-10-12T22:32:03.521 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.512+0000 7f6935ffb700 1 -- 172.21.15.195:0/3492779703 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f693c0263a0 con 0x7f69400f5410 2023-10-12T22:32:03.521 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.512+0000 7f6947464700 1 -- 172.21.15.195:0/3492779703 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6940045ad0 con 0x7f69400f5410 2023-10-12T22:32:03.521 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.512+0000 7f6935ffb700 1 --2- 172.21.15.195:0/3492779703 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f692c02ff60 0x7f692c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:03.521 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.512+0000 7f6935ffb700 1 -- 172.21.15.195:0/3492779703 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f693c041390 con 0x7f69400f5410 2023-10-12T22:32:03.521 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.513+0000 7f69449ff700 1 --2- 172.21.15.195:0/3492779703 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f692c02ff60 0x7f692c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:03.522 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.513+0000 7f69449ff700 1 --2- 172.21.15.195:0/3492779703 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f692c02ff60 0x7f692c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f69300096a0 tx=0x7f6930006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:03.522 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.517+0000 7f6935ffb700 1 -- 172.21.15.195:0/3492779703 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f693c01d380 con 0x7f69400f5410 2023-10-12T22:32:03.724 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.723+0000 7f6947464700 1 -- 172.21.15.195:0/3492779703 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69400fae60 con 0x7f69400f5410 2023-10-12T22:32:03.725 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.724+0000 7f6935ffb700 1 -- 172.21.15.195:0/3492779703 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f693c01cb10 con 0x7f69400f5410 2023-10-12T22:32:03.725 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:03.726 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:03.727 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.725+0000 7f6947464700 1 -- 172.21.15.195:0/3492779703 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f692c02ff60 msgr2=0x7f692c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:03.727 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.725+0000 7f6947464700 1 --2- 172.21.15.195:0/3492779703 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f692c02ff60 0x7f692c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f69300096a0 tx=0x7f6930006b40).stop 2023-10-12T22:32:03.727 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.725+0000 7f6947464700 1 -- 172.21.15.195:0/3492779703 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69400f5410 msgr2=0x7f6940105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:03.727 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.725+0000 7f6947464700 1 --2- 172.21.15.195:0/3492779703 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69400f5410 0x7f6940105630 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f693c013510 tx=0x7f693c00b040).stop 2023-10-12T22:32:03.728 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.725+0000 7f6947464700 1 -- 172.21.15.195:0/3492779703 shutdown_connections 2023-10-12T22:32:03.728 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.725+0000 7f6947464700 1 --2- 172.21.15.195:0/3492779703 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f692c02ff60 0x7f692c032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:03.728 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.726+0000 7f6947464700 1 --2- 172.21.15.195:0/3492779703 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f69400f5410 0x7f6940105630 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:03.728 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.726+0000 7f6947464700 1 -- 172.21.15.195:0/3492779703 >> 172.21.15.195:0/3492779703 conn(0x7f69400f0b70 msgr2=0x7f69400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:03.728 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.726+0000 7f6947464700 1 -- 172.21.15.195:0/3492779703 shutdown_connections 2023-10-12T22:32:03.728 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:03.726+0000 7f6947464700 1 -- 172.21.15.195:0/3492779703 wait complete. 2023-10-12T22:32:03.729 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:04.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:02.800763+0000 mgr.smithi131.efktma (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:04.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:03.725308+0000 mon.smithi131 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.195:0/3492779703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:05.141 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:05.141 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:06.326 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.324+0000 7f7dba7fc700 1 -- 172.21.15.195:0/2279528788 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7da40138a0 con 0x7f7dbc0f6510 2023-10-12T22:32:06.327 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.325+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2279528788 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7dbc0f6510 msgr2=0x7f7dbc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:06.327 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.325+0000 7f7dc1c30700 1 --2- 172.21.15.195:0/2279528788 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7dbc0f6510 0x7f7dbc0f6930 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f7da4009fb0 tx=0x7f7da4009b30).stop 2023-10-12T22:32:06.327 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.325+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2279528788 shutdown_connections 2023-10-12T22:32:06.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.325+0000 7f7dc1c30700 1 --2- 172.21.15.195:0/2279528788 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7dbc0f6510 0x7f7dbc0f6930 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:06.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.325+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2279528788 >> 172.21.15.195:0/2279528788 conn(0x7f7dbc0f1c70 msgr2=0x7f7dbc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:06.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.325+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2279528788 shutdown_connections 2023-10-12T22:32:06.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.325+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2279528788 wait complete. 2023-10-12T22:32:06.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.326+0000 7f7dc1c30700 1 Processor -- start 2023-10-12T22:32:06.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.326+0000 7f7dc1c30700 1 -- start start 2023-10-12T22:32:06.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.326+0000 7f7dc1c30700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7dbc0f6510 0x7f7dbc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:06.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.326+0000 7f7dc1c30700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc106c20 con 0x7f7dbc0f6510 2023-10-12T22:32:06.332 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.326+0000 7f7dbb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7dbc0f6510 0x7f7dbc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:06.332 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.326+0000 7f7dbb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7dbc0f6510 0x7f7dbc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:53798/0 (socket says 172.21.15.195:53798) 2023-10-12T22:32:06.332 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.326+0000 7f7dbb7fe700 1 -- 172.21.15.195:0/2094899847 learned_addr learned my addr 172.21.15.195:0/2094899847 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:06.333 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.327+0000 7f7dbb7fe700 1 -- 172.21.15.195:0/2094899847 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da4005040 con 0x7f7dbc0f6510 2023-10-12T22:32:06.333 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.327+0000 7f7dbb7fe700 1 --2- 172.21.15.195:0/2094899847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7dbc0f6510 0x7f7dbc106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7da4013510 tx=0x7f7da400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:06.333 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.327+0000 7f7db3fff700 1 -- 172.21.15.195:0/2094899847 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7da4009320 con 0x7f7dbc0f6510 2023-10-12T22:32:06.333 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.327+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2094899847 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dbc106e20 con 0x7f7dbc0f6510 2023-10-12T22:32:06.334 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.327+0000 7f7db3fff700 1 -- 172.21.15.195:0/2094899847 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7da4013f00 con 0x7f7dbc0f6510 2023-10-12T22:32:06.334 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.327+0000 7f7db3fff700 1 -- 172.21.15.195:0/2094899847 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7da4014d90 con 0x7f7dbc0f6510 2023-10-12T22:32:06.334 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.327+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2094899847 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dbc1072b0 con 0x7f7dbc0f6510 2023-10-12T22:32:06.335 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.328+0000 7f7db3fff700 1 -- 172.21.15.195:0/2094899847 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7da40263a0 con 0x7f7dbc0f6510 2023-10-12T22:32:06.335 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.328+0000 7f7db3fff700 1 --2- 172.21.15.195:0/2094899847 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7da802ff60 0x7f7da8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:06.335 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.328+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2094899847 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dbc045ad0 con 0x7f7dbc0f6510 2023-10-12T22:32:06.335 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.328+0000 7f7db3fff700 1 -- 172.21.15.195:0/2094899847 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7da40410c0 con 0x7f7dbc0f6510 2023-10-12T22:32:06.336 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.328+0000 7f7dbaffd700 1 --2- 172.21.15.195:0/2094899847 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7da802ff60 0x7f7da8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:06.336 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.329+0000 7f7dbaffd700 1 --2- 172.21.15.195:0/2094899847 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7da802ff60 0x7f7da8032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f7dac0096a0 tx=0x7f7dac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:06.337 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.333+0000 7f7db3fff700 1 -- 172.21.15.195:0/2094899847 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7da4035080 con 0x7f7dbc0f6510 2023-10-12T22:32:06.541 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.540+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2094899847 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7dbc0fc050 con 0x7f7dbc0f6510 2023-10-12T22:32:06.542 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.541+0000 7f7db3fff700 1 -- 172.21.15.195:0/2094899847 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7da4012920 con 0x7f7dbc0f6510 2023-10-12T22:32:06.542 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:06.543 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:06.544 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.542+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2094899847 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7da802ff60 msgr2=0x7f7da8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:06.544 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.542+0000 7f7dc1c30700 1 --2- 172.21.15.195:0/2094899847 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7da802ff60 0x7f7da8032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f7dac0096a0 tx=0x7f7dac006b40).stop 2023-10-12T22:32:06.544 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.543+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2094899847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7dbc0f6510 msgr2=0x7f7dbc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:06.544 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.543+0000 7f7dc1c30700 1 --2- 172.21.15.195:0/2094899847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7dbc0f6510 0x7f7dbc106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7da4013510 tx=0x7f7da400b040).stop 2023-10-12T22:32:06.545 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.543+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2094899847 shutdown_connections 2023-10-12T22:32:06.545 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.543+0000 7f7dc1c30700 1 --2- 172.21.15.195:0/2094899847 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7da802ff60 0x7f7da8032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:06.545 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.543+0000 7f7dc1c30700 1 --2- 172.21.15.195:0/2094899847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7dbc0f6510 0x7f7dbc106730 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:06.545 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.543+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2094899847 >> 172.21.15.195:0/2094899847 conn(0x7f7dbc0f1c70 msgr2=0x7f7dbc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:06.545 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.543+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2094899847 shutdown_connections 2023-10-12T22:32:06.546 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:06.543+0000 7f7dc1c30700 1 -- 172.21.15.195:0/2094899847 wait complete. 2023-10-12T22:32:06.546 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:06.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:04.801029+0000 mgr.smithi131.efktma (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:07.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:06.542211+0000 mon.smithi131 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.195:0/2094899847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:07.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:07.929 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:08.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:06.801367+0000 mgr.smithi131.efktma (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:09.134 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.132+0000 7fdc4d27d700 1 -- 172.21.15.195:0/976347622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc48060280 msgr2=0x7fdc480f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:09.134 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.132+0000 7fdc4d27d700 1 --2- 172.21.15.195:0/976347622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc48060280 0x7fdc480f90f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdc38009fb0 tx=0x7fdc38009b30).stop 2023-10-12T22:32:09.134 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.132+0000 7fdc4d27d700 1 -- 172.21.15.195:0/976347622 shutdown_connections 2023-10-12T22:32:09.134 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.132+0000 7fdc4d27d700 1 --2- 172.21.15.195:0/976347622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc48060280 0x7fdc480f90f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:09.135 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.132+0000 7fdc4d27d700 1 -- 172.21.15.195:0/976347622 >> 172.21.15.195:0/976347622 conn(0x7fdc480f0b70 msgr2=0x7fdc480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:09.135 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.132+0000 7fdc4d27d700 1 -- 172.21.15.195:0/976347622 shutdown_connections 2023-10-12T22:32:09.135 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.132+0000 7fdc4d27d700 1 -- 172.21.15.195:0/976347622 wait complete. 2023-10-12T22:32:09.137 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.133+0000 7fdc4d27d700 1 Processor -- start 2023-10-12T22:32:09.137 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.133+0000 7fdc4d27d700 1 -- start start 2023-10-12T22:32:09.137 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.133+0000 7fdc4d27d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc48060280 0x7fdc481066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:09.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.133+0000 7fdc4d27d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc48106bd0 con 0x7fdc48060280 2023-10-12T22:32:09.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.133+0000 7fdc46ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc48060280 0x7fdc481066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:09.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.133+0000 7fdc46ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc48060280 0x7fdc481066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:37026/0 (socket says 172.21.15.195:37026) 2023-10-12T22:32:09.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.133+0000 7fdc46ffd700 1 -- 172.21.15.195:0/347000465 learned_addr learned my addr 172.21.15.195:0/347000465 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:09.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.134+0000 7fdc46ffd700 1 -- 172.21.15.195:0/347000465 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc38005040 con 0x7fdc48060280 2023-10-12T22:32:09.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.134+0000 7fdc46ffd700 1 --2- 172.21.15.195:0/347000465 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc48060280 0x7fdc481066e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fdc38013510 tx=0x7fdc3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:09.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.134+0000 7fdc2f7fe700 1 -- 172.21.15.195:0/347000465 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc38009320 con 0x7fdc48060280 2023-10-12T22:32:09.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.134+0000 7fdc4d27d700 1 -- 172.21.15.195:0/347000465 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc48106dd0 con 0x7fdc48060280 2023-10-12T22:32:09.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.134+0000 7fdc2f7fe700 1 -- 172.21.15.195:0/347000465 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc38013f00 con 0x7fdc48060280 2023-10-12T22:32:09.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.134+0000 7fdc2f7fe700 1 -- 172.21.15.195:0/347000465 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc38014dc0 con 0x7fdc48060280 2023-10-12T22:32:09.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.134+0000 7fdc4d27d700 1 -- 172.21.15.195:0/347000465 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc48107260 con 0x7fdc48060280 2023-10-12T22:32:09.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.135+0000 7fdc4d27d700 1 -- 172.21.15.195:0/347000465 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc48045ad0 con 0x7fdc48060280 2023-10-12T22:32:09.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.135+0000 7fdc2f7fe700 1 -- 172.21.15.195:0/347000465 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc380095f0 con 0x7fdc48060280 2023-10-12T22:32:09.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.135+0000 7fdc2f7fe700 1 --2- 172.21.15.195:0/347000465 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdc3002ff60 0x7fdc30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:09.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.135+0000 7fdc2f7fe700 1 -- 172.21.15.195:0/347000465 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc38041020 con 0x7fdc48060280 2023-10-12T22:32:09.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.135+0000 7fdc467fc700 1 --2- 172.21.15.195:0/347000465 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdc3002ff60 0x7fdc30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:09.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.136+0000 7fdc467fc700 1 --2- 172.21.15.195:0/347000465 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdc3002ff60 0x7fdc30032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdc3c0096a0 tx=0x7fdc3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:09.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.139+0000 7fdc2f7fe700 1 -- 172.21.15.195:0/347000465 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc38014420 con 0x7fdc48060280 2023-10-12T22:32:09.345 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.344+0000 7fdc4d27d700 1 -- 172.21.15.195:0/347000465 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc481080b0 con 0x7fdc48060280 2023-10-12T22:32:09.346 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.344+0000 7fdc2f7fe700 1 -- 172.21.15.195:0/347000465 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdc3801c920 con 0x7fdc48060280 2023-10-12T22:32:09.346 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:09.346 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:09.348 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.346+0000 7fdc4d27d700 1 -- 172.21.15.195:0/347000465 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdc3002ff60 msgr2=0x7fdc30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:09.348 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.346+0000 7fdc4d27d700 1 --2- 172.21.15.195:0/347000465 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdc3002ff60 0x7fdc30032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdc3c0096a0 tx=0x7fdc3c006b40).stop 2023-10-12T22:32:09.348 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.346+0000 7fdc4d27d700 1 -- 172.21.15.195:0/347000465 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc48060280 msgr2=0x7fdc481066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:09.348 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.346+0000 7fdc4d27d700 1 --2- 172.21.15.195:0/347000465 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc48060280 0x7fdc481066e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fdc38013510 tx=0x7fdc3800b040).stop 2023-10-12T22:32:09.348 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.346+0000 7fdc4d27d700 1 -- 172.21.15.195:0/347000465 shutdown_connections 2023-10-12T22:32:09.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.346+0000 7fdc4d27d700 1 --2- 172.21.15.195:0/347000465 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdc3002ff60 0x7fdc30032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:09.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.346+0000 7fdc4d27d700 1 --2- 172.21.15.195:0/347000465 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc48060280 0x7fdc481066e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:09.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.346+0000 7fdc4d27d700 1 -- 172.21.15.195:0/347000465 >> 172.21.15.195:0/347000465 conn(0x7fdc480f0b70 msgr2=0x7fdc480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:09.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.346+0000 7fdc4d27d700 1 -- 172.21.15.195:0/347000465 shutdown_connections 2023-10-12T22:32:09.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:09.346+0000 7fdc4d27d700 1 -- 172.21.15.195:0/347000465 wait complete. 2023-10-12T22:32:09.350 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:10.733 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:10.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:10.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:08.801711+0000 mgr.smithi131.efktma (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:10.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:09.345877+0000 mon.smithi131 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.195:0/347000465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:11.950 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.925+0000 7f38ad66f700 1 -- 172.21.15.195:0/3732698098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38a80f51c0 msgr2=0x7f38a80f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:11.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.925+0000 7f38ad66f700 1 --2- 172.21.15.195:0/3732698098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38a80f51c0 0x7f38a80f55e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f3898009fb0 tx=0x7f3898009b30).stop 2023-10-12T22:32:11.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.925+0000 7f38ad66f700 1 -- 172.21.15.195:0/3732698098 shutdown_connections 2023-10-12T22:32:11.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.925+0000 7f38ad66f700 1 --2- 172.21.15.195:0/3732698098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38a80f51c0 0x7f38a80f55e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:11.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.925+0000 7f38ad66f700 1 -- 172.21.15.195:0/3732698098 >> 172.21.15.195:0/3732698098 conn(0x7f38a80f0920 msgr2=0x7f38a80f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:11.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.925+0000 7f38ad66f700 1 -- 172.21.15.195:0/3732698098 shutdown_connections 2023-10-12T22:32:11.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.925+0000 7f38ad66f700 1 -- 172.21.15.195:0/3732698098 wait complete. 2023-10-12T22:32:11.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.926+0000 7f38ad66f700 1 Processor -- start 2023-10-12T22:32:11.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.926+0000 7f38ad66f700 1 -- start start 2023-10-12T22:32:11.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.926+0000 7f38ad66f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38a80f51c0 0x7f38a8105440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:11.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.926+0000 7f38ad66f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a8105930 con 0x7f38a80f51c0 2023-10-12T22:32:11.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.926+0000 7f38a6ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38a80f51c0 0x7f38a8105440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:11.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.926+0000 7f38a6ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38a80f51c0 0x7f38a8105440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:37042/0 (socket says 172.21.15.195:37042) 2023-10-12T22:32:11.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.926+0000 7f38a6ffd700 1 -- 172.21.15.195:0/76767890 learned_addr learned my addr 172.21.15.195:0/76767890 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:11.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.927+0000 7f38a6ffd700 1 -- 172.21.15.195:0/76767890 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3898005040 con 0x7f38a80f51c0 2023-10-12T22:32:11.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.927+0000 7f38a6ffd700 1 --2- 172.21.15.195:0/76767890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38a80f51c0 0x7f38a8105440 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f3898013510 tx=0x7f389800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:11.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.927+0000 7f388f7fe700 1 -- 172.21.15.195:0/76767890 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3898009320 con 0x7f38a80f51c0 2023-10-12T22:32:11.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.927+0000 7f38ad66f700 1 -- 172.21.15.195:0/76767890 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a8105b30 con 0x7f38a80f51c0 2023-10-12T22:32:11.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.927+0000 7f388f7fe700 1 -- 172.21.15.195:0/76767890 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3898013f00 con 0x7f38a80f51c0 2023-10-12T22:32:11.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.927+0000 7f388f7fe700 1 -- 172.21.15.195:0/76767890 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3898014d90 con 0x7f38a80f51c0 2023-10-12T22:32:11.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.927+0000 7f38ad66f700 1 -- 172.21.15.195:0/76767890 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a8105fc0 con 0x7f38a80f51c0 2023-10-12T22:32:11.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.928+0000 7f38ad66f700 1 -- 172.21.15.195:0/76767890 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a8045ad0 con 0x7f38a80f51c0 2023-10-12T22:32:11.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.928+0000 7f388f7fe700 1 -- 172.21.15.195:0/76767890 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38980095f0 con 0x7f38a80f51c0 2023-10-12T22:32:11.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.928+0000 7f388f7fe700 1 --2- 172.21.15.195:0/76767890 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f389002ff60 0x7f3890032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:11.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.928+0000 7f388f7fe700 1 -- 172.21.15.195:0/76767890 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3898040fe0 con 0x7f38a80f51c0 2023-10-12T22:32:11.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.928+0000 7f38a67fc700 1 --2- 172.21.15.195:0/76767890 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f389002ff60 0x7f3890032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:11.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.928+0000 7f38a67fc700 1 --2- 172.21.15.195:0/76767890 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f389002ff60 0x7f3890032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f389c0096a0 tx=0x7f389c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:11.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:11.933+0000 7f388f7fe700 1 -- 172.21.15.195:0/76767890 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3898010230 con 0x7f38a80f51c0 2023-10-12T22:32:12.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:12.139+0000 7f38ad66f700 1 -- 172.21.15.195:0/76767890 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38a8106dd0 con 0x7f38a80f51c0 2023-10-12T22:32:12.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:12.140+0000 7f388f7fe700 1 -- 172.21.15.195:0/76767890 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3898010230 con 0x7f38a80f51c0 2023-10-12T22:32:12.141 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:12.142 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:12.143 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:12.141+0000 7f38ad66f700 1 -- 172.21.15.195:0/76767890 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f389002ff60 msgr2=0x7f3890032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:12.143 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:12.141+0000 7f38ad66f700 1 --2- 172.21.15.195:0/76767890 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f389002ff60 0x7f3890032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f389c0096a0 tx=0x7f389c006b40).stop 2023-10-12T22:32:12.143 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:12.141+0000 7f38ad66f700 1 -- 172.21.15.195:0/76767890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38a80f51c0 msgr2=0x7f38a8105440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:12.143 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:12.141+0000 7f38ad66f700 1 --2- 172.21.15.195:0/76767890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38a80f51c0 0x7f38a8105440 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f3898013510 tx=0x7f389800b040).stop 2023-10-12T22:32:12.144 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:12.141+0000 7f38ad66f700 1 -- 172.21.15.195:0/76767890 shutdown_connections 2023-10-12T22:32:12.144 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:12.142+0000 7f38ad66f700 1 --2- 172.21.15.195:0/76767890 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f389002ff60 0x7f3890032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:12.144 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:12.142+0000 7f38ad66f700 1 --2- 172.21.15.195:0/76767890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38a80f51c0 0x7f38a8105440 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:12.144 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:12.142+0000 7f38ad66f700 1 -- 172.21.15.195:0/76767890 >> 172.21.15.195:0/76767890 conn(0x7f38a80f0920 msgr2=0x7f38a80f14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:12.144 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:12.142+0000 7f38ad66f700 1 -- 172.21.15.195:0/76767890 shutdown_connections 2023-10-12T22:32:12.145 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:12.142+0000 7f38ad66f700 1 -- 172.21.15.195:0/76767890 wait complete. 2023-10-12T22:32:12.145 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:12.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:10.802069+0000 mgr.smithi131.efktma (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:12.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:12.141151+0000 mon.smithi131 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.195:0/76767890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:13.850 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:13.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:14.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:14 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:12.802470+0000 mgr.smithi131.efktma (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:15.041 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.038+0000 7f30c22ff700 1 -- 172.21.15.195:0/274509710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30bc0f6510 msgr2=0x7f30bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:15.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.038+0000 7f30c22ff700 1 --2- 172.21.15.195:0/274509710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30bc0f6510 0x7f30bc0f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f30ac009fb0 tx=0x7f30ac009b30).stop 2023-10-12T22:32:15.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.039+0000 7f30c22ff700 1 -- 172.21.15.195:0/274509710 shutdown_connections 2023-10-12T22:32:15.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.039+0000 7f30c22ff700 1 --2- 172.21.15.195:0/274509710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30bc0f6510 0x7f30bc0f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:15.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.039+0000 7f30c22ff700 1 -- 172.21.15.195:0/274509710 >> 172.21.15.195:0/274509710 conn(0x7f30bc0f1c70 msgr2=0x7f30bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:15.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.039+0000 7f30c22ff700 1 -- 172.21.15.195:0/274509710 shutdown_connections 2023-10-12T22:32:15.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.039+0000 7f30c22ff700 1 -- 172.21.15.195:0/274509710 wait complete. 2023-10-12T22:32:15.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.039+0000 7f30c22ff700 1 Processor -- start 2023-10-12T22:32:15.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.039+0000 7f30c22ff700 1 -- start start 2023-10-12T22:32:15.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.039+0000 7f30c22ff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30bc106840 0x7f30bc106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:15.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.039+0000 7f30c22ff700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc107150 con 0x7f30bc106840 2023-10-12T22:32:15.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.040+0000 7f30bbfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30bc106840 0x7f30bc106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:15.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.040+0000 7f30bbfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30bc106840 0x7f30bc106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:37062/0 (socket says 172.21.15.195:37062) 2023-10-12T22:32:15.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.040+0000 7f30bbfff700 1 -- 172.21.15.195:0/2773991881 learned_addr learned my addr 172.21.15.195:0/2773991881 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:15.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.040+0000 7f30bbfff700 1 -- 172.21.15.195:0/2773991881 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac005040 con 0x7f30bc106840 2023-10-12T22:32:15.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.040+0000 7f30bbfff700 1 --2- 172.21.15.195:0/2773991881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30bc106840 0x7f30bc106c60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f30ac009f80 tx=0x7f30ac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:15.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.040+0000 7f30b8ff9700 1 -- 172.21.15.195:0/2773991881 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30ac0136c0 con 0x7f30bc106840 2023-10-12T22:32:15.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.040+0000 7f30c22ff700 1 -- 172.21.15.195:0/2773991881 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30bc107350 con 0x7f30bc106840 2023-10-12T22:32:15.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.040+0000 7f30b8ff9700 1 -- 172.21.15.195:0/2773991881 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30ac013e70 con 0x7f30bc106840 2023-10-12T22:32:15.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.040+0000 7f30b8ff9700 1 -- 172.21.15.195:0/2773991881 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30ac014420 con 0x7f30bc106840 2023-10-12T22:32:15.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.040+0000 7f30c22ff700 1 -- 172.21.15.195:0/2773991881 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30bc107ea0 con 0x7f30bc106840 2023-10-12T22:32:15.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.041+0000 7f30b8ff9700 1 -- 172.21.15.195:0/2773991881 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30ac025060 con 0x7f30bc106840 2023-10-12T22:32:15.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.041+0000 7f30c22ff700 1 -- 172.21.15.195:0/2773991881 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30bc059100 con 0x7f30bc106840 2023-10-12T22:32:15.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.041+0000 7f30b8ff9700 1 --2- 172.21.15.195:0/2773991881 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f30a402ff60 0x7f30a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:15.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.041+0000 7f30b8ff9700 1 -- 172.21.15.195:0/2773991881 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30ac040e50 con 0x7f30bc106840 2023-10-12T22:32:15.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.041+0000 7f30bb7fe700 1 --2- 172.21.15.195:0/2773991881 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f30a402ff60 0x7f30a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:15.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.042+0000 7f30bb7fe700 1 --2- 172.21.15.195:0/2773991881 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f30a402ff60 0x7f30a4032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f30b00096a0 tx=0x7f30b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:15.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.046+0000 7f30b8ff9700 1 -- 172.21.15.195:0/2773991881 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30ac01cb30 con 0x7f30bc106840 2023-10-12T22:32:15.253 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.252+0000 7f30c22ff700 1 -- 172.21.15.195:0/2773991881 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30bc045ad0 con 0x7f30bc106840 2023-10-12T22:32:15.254 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.253+0000 7f30b8ff9700 1 -- 172.21.15.195:0/2773991881 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f30ac02d2d0 con 0x7f30bc106840 2023-10-12T22:32:15.255 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:15.255 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:15.256 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.254+0000 7f30c22ff700 1 -- 172.21.15.195:0/2773991881 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f30a402ff60 msgr2=0x7f30a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:15.256 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.254+0000 7f30c22ff700 1 --2- 172.21.15.195:0/2773991881 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f30a402ff60 0x7f30a4032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f30b00096a0 tx=0x7f30b0006b40).stop 2023-10-12T22:32:15.256 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.254+0000 7f30c22ff700 1 -- 172.21.15.195:0/2773991881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30bc106840 msgr2=0x7f30bc106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:15.257 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.254+0000 7f30c22ff700 1 --2- 172.21.15.195:0/2773991881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30bc106840 0x7f30bc106c60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f30ac009f80 tx=0x7f30ac009b30).stop 2023-10-12T22:32:15.257 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.254+0000 7f30c22ff700 1 -- 172.21.15.195:0/2773991881 shutdown_connections 2023-10-12T22:32:15.257 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.254+0000 7f30c22ff700 1 --2- 172.21.15.195:0/2773991881 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f30a402ff60 0x7f30a4032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:15.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.254+0000 7f30c22ff700 1 --2- 172.21.15.195:0/2773991881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30bc106840 0x7f30bc106c60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:15.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.254+0000 7f30c22ff700 1 -- 172.21.15.195:0/2773991881 >> 172.21.15.195:0/2773991881 conn(0x7f30bc0f1c70 msgr2=0x7f30bc0f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:15.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.254+0000 7f30c22ff700 1 -- 172.21.15.195:0/2773991881 shutdown_connections 2023-10-12T22:32:15.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:15.254+0000 7f30c22ff700 1 -- 172.21.15.195:0/2773991881 wait complete. 2023-10-12T22:32:15.259 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:16.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:15 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:14.802762+0000 mgr.smithi131.efktma (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:16.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:15 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:15.254018+0000 mon.smithi131 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.195:0/2773991881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:16.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:16.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:18.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:17 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:16.803074+0000 mgr.smithi131.efktma (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:19.027 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.997+0000 7f62051e8700 1 -- 172.21.15.195:0/3667064785 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62000f62a0 msgr2=0x7f62000f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:19.028 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.997+0000 7f62051e8700 1 --2- 172.21.15.195:0/3667064785 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62000f62a0 0x7f62000f66c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f61f0009fb0 tx=0x7f61f0009b30).stop 2023-10-12T22:32:19.028 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.998+0000 7f62051e8700 1 -- 172.21.15.195:0/3667064785 shutdown_connections 2023-10-12T22:32:19.028 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.998+0000 7f62051e8700 1 --2- 172.21.15.195:0/3667064785 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62000f62a0 0x7f62000f66c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:19.028 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.998+0000 7f62051e8700 1 -- 172.21.15.195:0/3667064785 >> 172.21.15.195:0/3667064785 conn(0x7f62000f1aa0 msgr2=0x7f62000f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:19.029 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.998+0000 7f62051e8700 1 -- 172.21.15.195:0/3667064785 shutdown_connections 2023-10-12T22:32:19.029 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.998+0000 7f62051e8700 1 -- 172.21.15.195:0/3667064785 wait complete. 2023-10-12T22:32:19.029 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.998+0000 7f62051e8700 1 Processor -- start 2023-10-12T22:32:19.029 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.999+0000 7f62051e8700 1 -- start start 2023-10-12T22:32:19.029 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.999+0000 7f62051e8700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62000f62a0 0x7f62001064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:19.029 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.999+0000 7f62051e8700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62001069a0 con 0x7f62000f62a0 2023-10-12T22:32:19.030 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.999+0000 7f61fffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62000f62a0 0x7f62001064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:19.030 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.999+0000 7f61fffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62000f62a0 0x7f62001064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:36918/0 (socket says 172.21.15.195:36918) 2023-10-12T22:32:19.030 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:18.999+0000 7f61fffff700 1 -- 172.21.15.195:0/3916864213 learned_addr learned my addr 172.21.15.195:0/3916864213 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:19.030 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.000+0000 7f61fffff700 1 -- 172.21.15.195:0/3916864213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f0005040 con 0x7f62000f62a0 2023-10-12T22:32:19.030 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.000+0000 7f61fffff700 1 --2- 172.21.15.195:0/3916864213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62000f62a0 0x7f62001064b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f61f0013510 tx=0x7f61f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:19.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.000+0000 7f61fcff9700 1 -- 172.21.15.195:0/3916864213 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61f0009320 con 0x7f62000f62a0 2023-10-12T22:32:19.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.000+0000 7f61fcff9700 1 -- 172.21.15.195:0/3916864213 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61f0013f00 con 0x7f62000f62a0 2023-10-12T22:32:19.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.000+0000 7f62051e8700 1 -- 172.21.15.195:0/3916864213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6200106ba0 con 0x7f62000f62a0 2023-10-12T22:32:19.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.000+0000 7f61fcff9700 1 -- 172.21.15.195:0/3916864213 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61f0014dc0 con 0x7f62000f62a0 2023-10-12T22:32:19.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.000+0000 7f62051e8700 1 -- 172.21.15.195:0/3916864213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6200107030 con 0x7f62000f62a0 2023-10-12T22:32:19.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.001+0000 7f61fcff9700 1 -- 172.21.15.195:0/3916864213 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61f001d3d0 con 0x7f62000f62a0 2023-10-12T22:32:19.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.001+0000 7f62051e8700 1 -- 172.21.15.195:0/3916864213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6200045ad0 con 0x7f62000f62a0 2023-10-12T22:32:19.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.001+0000 7f61fcff9700 1 --2- 172.21.15.195:0/3916864213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f61e802ff60 0x7f61e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:19.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.001+0000 7f61fcff9700 1 -- 172.21.15.195:0/3916864213 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61f0041380 con 0x7f62000f62a0 2023-10-12T22:32:19.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.001+0000 7f61ff7fe700 1 --2- 172.21.15.195:0/3916864213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f61e802ff60 0x7f61e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:19.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.002+0000 7f61ff7fe700 1 --2- 172.21.15.195:0/3916864213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f61e802ff60 0x7f61e8032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f61f40096a0 tx=0x7f61f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:19.033 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.006+0000 7f61fcff9700 1 -- 172.21.15.195:0/3916864213 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61f0012510 con 0x7f62000f62a0 2023-10-12T22:32:19.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.215+0000 7f62051e8700 1 -- 172.21.15.195:0/3916864213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6200107ee0 con 0x7f62000f62a0 2023-10-12T22:32:19.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.215+0000 7f61fcff9700 1 -- 172.21.15.195:0/3916864213 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f61f0012330 con 0x7f62000f62a0 2023-10-12T22:32:19.217 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:19.218 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:19.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.217+0000 7f62051e8700 1 -- 172.21.15.195:0/3916864213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f61e802ff60 msgr2=0x7f61e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:19.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.217+0000 7f62051e8700 1 --2- 172.21.15.195:0/3916864213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f61e802ff60 0x7f61e8032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f61f40096a0 tx=0x7f61f4006b40).stop 2023-10-12T22:32:19.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.217+0000 7f62051e8700 1 -- 172.21.15.195:0/3916864213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62000f62a0 msgr2=0x7f62001064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:19.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.217+0000 7f62051e8700 1 --2- 172.21.15.195:0/3916864213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62000f62a0 0x7f62001064b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f61f0013510 tx=0x7f61f000b040).stop 2023-10-12T22:32:19.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.217+0000 7f62051e8700 1 -- 172.21.15.195:0/3916864213 shutdown_connections 2023-10-12T22:32:19.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.217+0000 7f62051e8700 1 --2- 172.21.15.195:0/3916864213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f61e802ff60 0x7f61e8032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:19.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.217+0000 7f62051e8700 1 --2- 172.21.15.195:0/3916864213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62000f62a0 0x7f62001064b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:19.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.217+0000 7f62051e8700 1 -- 172.21.15.195:0/3916864213 >> 172.21.15.195:0/3916864213 conn(0x7f62000f1aa0 msgr2=0x7f62000f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:19.223 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.217+0000 7f62051e8700 1 -- 172.21.15.195:0/3916864213 shutdown_connections 2023-10-12T22:32:19.223 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:19.217+0000 7f62051e8700 1 -- 172.21.15.195:0/3916864213 wait complete. 2023-10-12T22:32:19.223 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:20.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:19 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:18.803304+0000 mgr.smithi131.efktma (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:20.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:19 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:19.216754+0000 mon.smithi131 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.195:0/3916864213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:22.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:21 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:20.803728+0000 mgr.smithi131.efktma (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:22.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:22.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:23.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:22.929959+0000 mon.smithi131 (mon.0) 128 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/mirror_snapshot_schedule"}]: dispatch 2023-10-12T22:32:23.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:22.937835+0000 mon.smithi131 (mon.0) 129 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/trash_purge_schedule"}]: dispatch 2023-10-12T22:32:24.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:22.804082+0000 mgr.smithi131.efktma (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:26.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:25 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:24.804336+0000 mgr.smithi131.efktma (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:26.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.494+0000 7f818fc60700 1 -- 172.21.15.195:0/2472856680 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81880f6480 msgr2=0x7f81880f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:26.498 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.494+0000 7f818fc60700 1 --2- 172.21.15.195:0/2472856680 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81880f6480 0x7f81880f68a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f8178009fb0 tx=0x7f8178009b30).stop 2023-10-12T22:32:26.498 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.494+0000 7f818fc60700 1 -- 172.21.15.195:0/2472856680 shutdown_connections 2023-10-12T22:32:26.498 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.494+0000 7f818fc60700 1 --2- 172.21.15.195:0/2472856680 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81880f6480 0x7f81880f68a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:26.498 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.494+0000 7f818fc60700 1 -- 172.21.15.195:0/2472856680 >> 172.21.15.195:0/2472856680 conn(0x7f81880f1c00 msgr2=0x7f81880f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:26.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.495+0000 7f818fc60700 1 -- 172.21.15.195:0/2472856680 shutdown_connections 2023-10-12T22:32:26.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.495+0000 7f818fc60700 1 -- 172.21.15.195:0/2472856680 wait complete. 2023-10-12T22:32:26.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.495+0000 7f818fc60700 1 Processor -- start 2023-10-12T22:32:26.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.495+0000 7f818fc60700 1 -- start start 2023-10-12T22:32:26.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.495+0000 7f818fc60700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81880f6480 0x7f81881066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:26.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.495+0000 7f818fc60700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8188106bc0 con 0x7f81880f6480 2023-10-12T22:32:26.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.495+0000 7f818d9fc700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81880f6480 0x7f81881066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:26.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.495+0000 7f818d9fc700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81880f6480 0x7f81881066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:36942/0 (socket says 172.21.15.195:36942) 2023-10-12T22:32:26.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.495+0000 7f818d9fc700 1 -- 172.21.15.195:0/1924475532 learned_addr learned my addr 172.21.15.195:0/1924475532 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:26.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.496+0000 7f818d9fc700 1 -- 172.21.15.195:0/1924475532 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8178005040 con 0x7f81880f6480 2023-10-12T22:32:26.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.496+0000 7f818d9fc700 1 --2- 172.21.15.195:0/1924475532 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81880f6480 0x7f81881066d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f8178013510 tx=0x7f817800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:26.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.496+0000 7f817e7fc700 1 -- 172.21.15.195:0/1924475532 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8178009320 con 0x7f81880f6480 2023-10-12T22:32:26.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.496+0000 7f817e7fc700 1 -- 172.21.15.195:0/1924475532 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8178013f00 con 0x7f81880f6480 2023-10-12T22:32:26.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.496+0000 7f818fc60700 1 -- 172.21.15.195:0/1924475532 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8188106dc0 con 0x7f81880f6480 2023-10-12T22:32:26.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.496+0000 7f817e7fc700 1 -- 172.21.15.195:0/1924475532 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8178014d90 con 0x7f81880f6480 2023-10-12T22:32:26.502 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.496+0000 7f818fc60700 1 -- 172.21.15.195:0/1924475532 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8188107250 con 0x7f81880f6480 2023-10-12T22:32:26.503 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.497+0000 7f817e7fc700 1 -- 172.21.15.195:0/1924475532 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8178014420 con 0x7f81880f6480 2023-10-12T22:32:26.503 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.497+0000 7f818fc60700 1 -- 172.21.15.195:0/1924475532 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8188045ad0 con 0x7f81880f6480 2023-10-12T22:32:26.503 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.497+0000 7f817e7fc700 1 --2- 172.21.15.195:0/1924475532 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f817402ff60 0x7f8174032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:26.503 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.497+0000 7f817e7fc700 1 -- 172.21.15.195:0/1924475532 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f81780410c0 con 0x7f81880f6480 2023-10-12T22:32:26.504 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.497+0000 7f818d1fb700 1 --2- 172.21.15.195:0/1924475532 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f817402ff60 0x7f8174032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:26.504 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.498+0000 7f818d1fb700 1 --2- 172.21.15.195:0/1924475532 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f817402ff60 0x7f8174032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f81840096a0 tx=0x7f8184006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:26.504 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.502+0000 7f817e7fc700 1 -- 172.21.15.195:0/1924475532 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81780095f0 con 0x7f81880f6480 2023-10-12T22:32:26.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.712+0000 7f818fc60700 1 -- 172.21.15.195:0/1924475532 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8188108100 con 0x7f81880f6480 2023-10-12T22:32:26.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.713+0000 7f817e7fc700 1 -- 172.21.15.195:0/1924475532 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f817801db70 con 0x7f81880f6480 2023-10-12T22:32:26.715 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:26.715 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:26.716 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.714+0000 7f818fc60700 1 -- 172.21.15.195:0/1924475532 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f817402ff60 msgr2=0x7f8174032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:26.716 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.714+0000 7f818fc60700 1 --2- 172.21.15.195:0/1924475532 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f817402ff60 0x7f8174032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f81840096a0 tx=0x7f8184006b40).stop 2023-10-12T22:32:26.716 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.714+0000 7f818fc60700 1 -- 172.21.15.195:0/1924475532 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81880f6480 msgr2=0x7f81881066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:26.716 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.714+0000 7f818fc60700 1 --2- 172.21.15.195:0/1924475532 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81880f6480 0x7f81881066d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f8178013510 tx=0x7f817800b040).stop 2023-10-12T22:32:26.717 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.714+0000 7f818fc60700 1 -- 172.21.15.195:0/1924475532 shutdown_connections 2023-10-12T22:32:26.717 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.714+0000 7f818fc60700 1 --2- 172.21.15.195:0/1924475532 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f817402ff60 0x7f8174032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:26.717 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.714+0000 7f818fc60700 1 --2- 172.21.15.195:0/1924475532 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81880f6480 0x7f81881066d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:26.717 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.715+0000 7f818fc60700 1 -- 172.21.15.195:0/1924475532 >> 172.21.15.195:0/1924475532 conn(0x7f81880f1c00 msgr2=0x7f81880f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:26.717 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.715+0000 7f818fc60700 1 -- 172.21.15.195:0/1924475532 shutdown_connections 2023-10-12T22:32:26.718 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:26.715+0000 7f818fc60700 1 -- 172.21.15.195:0/1924475532 wait complete. 2023-10-12T22:32:26.718 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:27.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:26 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:26.714324+0000 mon.smithi131 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.195:0/1924475532' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:28.117 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:28.118 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:28.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:27 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:26.804649+0000 mgr.smithi131.efktma (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:29.255 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.254+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1164755727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd19c0f5410 msgr2=0x7fd19c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:29.256 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.254+0000 7fd1a2fc9700 1 --2- 172.21.15.195:0/1164755727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd19c0f5410 0x7fd19c0f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fd190009fb0 tx=0x7fd190009b30).stop 2023-10-12T22:32:29.256 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.254+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1164755727 shutdown_connections 2023-10-12T22:32:29.256 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.254+0000 7fd1a2fc9700 1 --2- 172.21.15.195:0/1164755727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd19c0f5410 0x7fd19c0f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:29.256 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.254+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1164755727 >> 172.21.15.195:0/1164755727 conn(0x7fd19c0f0b70 msgr2=0x7fd19c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:29.257 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.254+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1164755727 shutdown_connections 2023-10-12T22:32:29.257 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.254+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1164755727 wait complete. 2023-10-12T22:32:29.257 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.254+0000 7fd1a2fc9700 1 Processor -- start 2023-10-12T22:32:29.257 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.254+0000 7fd1a2fc9700 1 -- start start 2023-10-12T22:32:29.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.255+0000 7fd1a2fc9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd19c0f5410 0x7fd19c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:29.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.255+0000 7fd1a2fc9700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c105b20 con 0x7fd19c0f5410 2023-10-12T22:32:29.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.255+0000 7fd1a0d65700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd19c0f5410 0x7fd19c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:29.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.255+0000 7fd1a0d65700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd19c0f5410 0x7fd19c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38280/0 (socket says 172.21.15.195:38280) 2023-10-12T22:32:29.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.255+0000 7fd1a0d65700 1 -- 172.21.15.195:0/1717709154 learned_addr learned my addr 172.21.15.195:0/1717709154 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:29.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.256+0000 7fd1a0d65700 1 -- 172.21.15.195:0/1717709154 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd190005040 con 0x7fd19c0f5410 2023-10-12T22:32:29.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.256+0000 7fd1a0d65700 1 --2- 172.21.15.195:0/1717709154 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd19c0f5410 0x7fd19c105630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd190013510 tx=0x7fd19000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:29.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.256+0000 7fd1997fa700 1 -- 172.21.15.195:0/1717709154 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd190009320 con 0x7fd19c0f5410 2023-10-12T22:32:29.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.256+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1717709154 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19c105d20 con 0x7fd19c0f5410 2023-10-12T22:32:29.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.256+0000 7fd1997fa700 1 -- 172.21.15.195:0/1717709154 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd190013f00 con 0x7fd19c0f5410 2023-10-12T22:32:29.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.256+0000 7fd1997fa700 1 -- 172.21.15.195:0/1717709154 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd190014d90 con 0x7fd19c0f5410 2023-10-12T22:32:29.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.256+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1717709154 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19c1061b0 con 0x7fd19c0f5410 2023-10-12T22:32:29.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.257+0000 7fd1997fa700 1 -- 172.21.15.195:0/1717709154 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1900263a0 con 0x7fd19c0f5410 2023-10-12T22:32:29.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.257+0000 7fd1997fa700 1 --2- 172.21.15.195:0/1717709154 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd18c02ff60 0x7fd18c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:29.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.257+0000 7fd1997fa700 1 -- 172.21.15.195:0/1717709154 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd1900410f0 con 0x7fd19c0f5410 2023-10-12T22:32:29.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.257+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1717709154 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19c045ad0 con 0x7fd19c0f5410 2023-10-12T22:32:29.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.257+0000 7fd19bfff700 1 --2- 172.21.15.195:0/1717709154 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd18c02ff60 0x7fd18c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:29.263 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.258+0000 7fd19bfff700 1 --2- 172.21.15.195:0/1717709154 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd18c02ff60 0x7fd18c032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fd1880096a0 tx=0x7fd188006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:29.263 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.262+0000 7fd1997fa700 1 -- 172.21.15.195:0/1717709154 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd190012920 con 0x7fd19c0f5410 2023-10-12T22:32:29.470 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.469+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1717709154 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd19c0faf50 con 0x7fd19c0f5410 2023-10-12T22:32:29.471 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.470+0000 7fd1997fa700 1 -- 172.21.15.195:0/1717709154 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd19002b020 con 0x7fd19c0f5410 2023-10-12T22:32:29.472 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:29.472 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:29.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.471+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1717709154 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd18c02ff60 msgr2=0x7fd18c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:29.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.471+0000 7fd1a2fc9700 1 --2- 172.21.15.195:0/1717709154 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd18c02ff60 0x7fd18c032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fd1880096a0 tx=0x7fd188006b40).stop 2023-10-12T22:32:29.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.471+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1717709154 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd19c0f5410 msgr2=0x7fd19c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:29.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.471+0000 7fd1a2fc9700 1 --2- 172.21.15.195:0/1717709154 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd19c0f5410 0x7fd19c105630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd190013510 tx=0x7fd19000b040).stop 2023-10-12T22:32:29.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.471+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1717709154 shutdown_connections 2023-10-12T22:32:29.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.471+0000 7fd1a2fc9700 1 --2- 172.21.15.195:0/1717709154 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd18c02ff60 0x7fd18c032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:29.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.471+0000 7fd1a2fc9700 1 --2- 172.21.15.195:0/1717709154 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd19c0f5410 0x7fd19c105630 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:29.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.471+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1717709154 >> 172.21.15.195:0/1717709154 conn(0x7fd19c0f0b70 msgr2=0x7fd19c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:29.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.472+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1717709154 shutdown_connections 2023-10-12T22:32:29.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:29.472+0000 7fd1a2fc9700 1 -- 172.21.15.195:0/1717709154 wait complete. 2023-10-12T22:32:29.478 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:30.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:28.805059+0000 mgr.smithi131.efktma (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:30.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:29.471243+0000 mon.smithi131 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.195:0/1717709154' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:30.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:30.866 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:32.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.008+0000 7ff8185d8700 1 -- 172.21.15.195:0/2595444201 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8100f5410 msgr2=0x7ff8100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:32.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.008+0000 7ff8185d8700 1 --2- 172.21.15.195:0/2595444201 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8100f5410 0x7ff8100f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff800009fb0 tx=0x7ff800009b30).stop 2023-10-12T22:32:32.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.008+0000 7ff8185d8700 1 -- 172.21.15.195:0/2595444201 shutdown_connections 2023-10-12T22:32:32.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.008+0000 7ff8185d8700 1 --2- 172.21.15.195:0/2595444201 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8100f5410 0x7ff8100f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:32.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.008+0000 7ff8185d8700 1 -- 172.21.15.195:0/2595444201 >> 172.21.15.195:0/2595444201 conn(0x7ff8100f0b70 msgr2=0x7ff8100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:32.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.008+0000 7ff8185d8700 1 -- 172.21.15.195:0/2595444201 shutdown_connections 2023-10-12T22:32:32.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.008+0000 7ff8185d8700 1 -- 172.21.15.195:0/2595444201 wait complete. 2023-10-12T22:32:32.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.009+0000 7ff8185d8700 1 Processor -- start 2023-10-12T22:32:32.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.009+0000 7ff8185d8700 1 -- start start 2023-10-12T22:32:32.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.009+0000 7ff8185d8700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8100f5410 0x7ff81010b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:32.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.009+0000 7ff8185d8700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff810107210 con 0x7ff8100f5410 2023-10-12T22:32:32.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.009+0000 7ff816374700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8100f5410 0x7ff81010b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:32.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.009+0000 7ff816374700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8100f5410 0x7ff81010b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38290/0 (socket says 172.21.15.195:38290) 2023-10-12T22:32:32.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.009+0000 7ff816374700 1 -- 172.21.15.195:0/2533563336 learned_addr learned my addr 172.21.15.195:0/2533563336 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:32.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.010+0000 7ff816374700 1 -- 172.21.15.195:0/2533563336 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800005040 con 0x7ff8100f5410 2023-10-12T22:32:32.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.010+0000 7ff816374700 1 --2- 172.21.15.195:0/2533563336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8100f5410 0x7ff81010b820 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff800013510 tx=0x7ff80000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:32.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.010+0000 7ff806ffd700 1 -- 172.21.15.195:0/2533563336 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff800009320 con 0x7ff8100f5410 2023-10-12T22:32:32.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.010+0000 7ff8185d8700 1 -- 172.21.15.195:0/2533563336 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff810060220 con 0x7ff8100f5410 2023-10-12T22:32:32.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.010+0000 7ff806ffd700 1 -- 172.21.15.195:0/2533563336 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff800013f00 con 0x7ff8100f5410 2023-10-12T22:32:32.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.010+0000 7ff8185d8700 1 -- 172.21.15.195:0/2533563336 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81010bd10 con 0x7ff8100f5410 2023-10-12T22:32:32.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.010+0000 7ff806ffd700 1 -- 172.21.15.195:0/2533563336 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff800014d90 con 0x7ff8100f5410 2023-10-12T22:32:32.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.011+0000 7ff806ffd700 1 -- 172.21.15.195:0/2533563336 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8000263a0 con 0x7ff8100f5410 2023-10-12T22:32:32.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.011+0000 7ff806ffd700 1 --2- 172.21.15.195:0/2533563336 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff7fc02ff60 0x7ff7fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:32.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.011+0000 7ff806ffd700 1 -- 172.21.15.195:0/2533563336 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff800041010 con 0x7ff8100f5410 2023-10-12T22:32:32.017 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.011+0000 7ff8185d8700 1 -- 172.21.15.195:0/2533563336 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff810045ad0 con 0x7ff8100f5410 2023-10-12T22:32:32.017 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.011+0000 7ff815b73700 1 --2- 172.21.15.195:0/2533563336 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff7fc02ff60 0x7ff7fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:32.017 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.011+0000 7ff815b73700 1 --2- 172.21.15.195:0/2533563336 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff7fc02ff60 0x7ff7fc032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff80c0096a0 tx=0x7ff80c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:32.017 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.016+0000 7ff806ffd700 1 -- 172.21.15.195:0/2533563336 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff800012920 con 0x7ff8100f5410 2023-10-12T22:32:32.224 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.223+0000 7ff8185d8700 1 -- 172.21.15.195:0/2533563336 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8100fae60 con 0x7ff8100f5410 2023-10-12T22:32:32.225 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.225+0000 7ff806ffd700 1 -- 172.21.15.195:0/2533563336 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff8100fae60 con 0x7ff8100f5410 2023-10-12T22:32:32.225 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:32.225 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:32.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.225+0000 7ff8185d8700 1 -- 172.21.15.195:0/2533563336 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff7fc02ff60 msgr2=0x7ff7fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:32.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.225+0000 7ff8185d8700 1 --2- 172.21.15.195:0/2533563336 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff7fc02ff60 0x7ff7fc032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff80c0096a0 tx=0x7ff80c006b40).stop 2023-10-12T22:32:32.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.225+0000 7ff8185d8700 1 -- 172.21.15.195:0/2533563336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8100f5410 msgr2=0x7ff81010b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:32.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.225+0000 7ff8185d8700 1 --2- 172.21.15.195:0/2533563336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8100f5410 0x7ff81010b820 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff800013510 tx=0x7ff80000b040).stop 2023-10-12T22:32:32.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.225+0000 7ff8185d8700 1 -- 172.21.15.195:0/2533563336 shutdown_connections 2023-10-12T22:32:32.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.225+0000 7ff8185d8700 1 --2- 172.21.15.195:0/2533563336 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff7fc02ff60 0x7ff7fc032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:32.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.225+0000 7ff8185d8700 1 --2- 172.21.15.195:0/2533563336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8100f5410 0x7ff81010b820 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:32.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.225+0000 7ff8185d8700 1 -- 172.21.15.195:0/2533563336 >> 172.21.15.195:0/2533563336 conn(0x7ff8100f0b70 msgr2=0x7ff8100f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:32.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.225+0000 7ff8185d8700 1 -- 172.21.15.195:0/2533563336 shutdown_connections 2023-10-12T22:32:32.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:32.226+0000 7ff8185d8700 1 -- 172.21.15.195:0/2533563336 wait complete. 2023-10-12T22:32:32.229 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:32.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:31 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:30.805409+0000 mgr.smithi131.efktma (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:33.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:32.225094+0000 mon.smithi131 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.195:0/2533563336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:33.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:33.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:34.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:33 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:32.805690+0000 mgr.smithi131.efktma (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.741+0000 7f2bba650700 1 -- 172.21.15.195:0/3106778017 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2bb40f5410 msgr2=0x7f2bb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.741+0000 7f2bba650700 1 --2- 172.21.15.195:0/3106778017 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2bb40f5410 0x7f2bb40f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f2ba4009fb0 tx=0x7f2ba4009b30).stop 2023-10-12T22:32:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.741+0000 7f2bba650700 1 -- 172.21.15.195:0/3106778017 shutdown_connections 2023-10-12T22:32:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.741+0000 7f2bba650700 1 --2- 172.21.15.195:0/3106778017 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2bb40f5410 0x7f2bb40f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.741+0000 7f2bba650700 1 -- 172.21.15.195:0/3106778017 >> 172.21.15.195:0/3106778017 conn(0x7f2bb40f0b70 msgr2=0x7f2bb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.741+0000 7f2bba650700 1 -- 172.21.15.195:0/3106778017 shutdown_connections 2023-10-12T22:32:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.741+0000 7f2bba650700 1 -- 172.21.15.195:0/3106778017 wait complete. 2023-10-12T22:32:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.742+0000 7f2bba650700 1 Processor -- start 2023-10-12T22:32:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.742+0000 7f2bba650700 1 -- start start 2023-10-12T22:32:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.742+0000 7f2bba650700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2bb40f5410 0x7f2bb4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.742+0000 7f2bba650700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb4105b20 con 0x7f2bb40f5410 2023-10-12T22:32:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.742+0000 7f2bb3fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2bb40f5410 0x7f2bb4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.742+0000 7f2bb3fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2bb40f5410 0x7f2bb4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38310/0 (socket says 172.21.15.195:38310) 2023-10-12T22:32:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.742+0000 7f2bb3fff700 1 -- 172.21.15.195:0/3079590899 learned_addr learned my addr 172.21.15.195:0/3079590899 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.743+0000 7f2bb3fff700 1 -- 172.21.15.195:0/3079590899 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba4005040 con 0x7f2bb40f5410 2023-10-12T22:32:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.743+0000 7f2bb3fff700 1 --2- 172.21.15.195:0/3079590899 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2bb40f5410 0x7f2bb4105630 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2ba4013510 tx=0x7f2ba400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.743+0000 7f2bb0ff9700 1 -- 172.21.15.195:0/3079590899 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ba4009320 con 0x7f2bb40f5410 2023-10-12T22:32:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.743+0000 7f2bba650700 1 -- 172.21.15.195:0/3079590899 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb4105d20 con 0x7f2bb40f5410 2023-10-12T22:32:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.743+0000 7f2bb0ff9700 1 -- 172.21.15.195:0/3079590899 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ba4013f00 con 0x7f2bb40f5410 2023-10-12T22:32:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.743+0000 7f2bb0ff9700 1 -- 172.21.15.195:0/3079590899 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ba4014d90 con 0x7f2bb40f5410 2023-10-12T22:32:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.743+0000 7f2bba650700 1 -- 172.21.15.195:0/3079590899 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb41061b0 con 0x7f2bb40f5410 2023-10-12T22:32:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.744+0000 7f2bb0ff9700 1 -- 172.21.15.195:0/3079590899 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ba40263a0 con 0x7f2bb40f5410 2023-10-12T22:32:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.744+0000 7f2bba650700 1 -- 172.21.15.195:0/3079590899 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb4045ad0 con 0x7f2bb40f5410 2023-10-12T22:32:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.744+0000 7f2bb0ff9700 1 --2- 172.21.15.195:0/3079590899 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2b9c02ff60 0x7f2b9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.744+0000 7f2bb0ff9700 1 -- 172.21.15.195:0/3079590899 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ba40412c0 con 0x7f2bb40f5410 2023-10-12T22:32:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.744+0000 7f2bb37fe700 1 --2- 172.21.15.195:0/3079590899 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2b9c02ff60 0x7f2b9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.745+0000 7f2bb37fe700 1 --2- 172.21.15.195:0/3079590899 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2b9c02ff60 0x7f2b9c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f2ba80096a0 tx=0x7f2ba8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:34.751 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.749+0000 7f2bb0ff9700 1 -- 172.21.15.195:0/3079590899 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ba401ccf0 con 0x7f2bb40f5410 2023-10-12T22:32:34.958 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.957+0000 7f2bba650700 1 -- 172.21.15.195:0/3079590899 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bb40fae60 con 0x7f2bb40f5410 2023-10-12T22:32:34.959 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.958+0000 7f2bb0ff9700 1 -- 172.21.15.195:0/3079590899 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2ba401cb10 con 0x7f2bb40f5410 2023-10-12T22:32:34.960 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:34.960 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:34.962 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.959+0000 7f2bba650700 1 -- 172.21.15.195:0/3079590899 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2b9c02ff60 msgr2=0x7f2b9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:34.962 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.959+0000 7f2bba650700 1 --2- 172.21.15.195:0/3079590899 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2b9c02ff60 0x7f2b9c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f2ba80096a0 tx=0x7f2ba8006b40).stop 2023-10-12T22:32:34.962 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.959+0000 7f2bba650700 1 -- 172.21.15.195:0/3079590899 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2bb40f5410 msgr2=0x7f2bb4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:34.963 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.959+0000 7f2bba650700 1 --2- 172.21.15.195:0/3079590899 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2bb40f5410 0x7f2bb4105630 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2ba4013510 tx=0x7f2ba400b040).stop 2023-10-12T22:32:34.963 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.959+0000 7f2bba650700 1 -- 172.21.15.195:0/3079590899 shutdown_connections 2023-10-12T22:32:34.964 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.959+0000 7f2bba650700 1 --2- 172.21.15.195:0/3079590899 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2b9c02ff60 0x7f2b9c032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:34.964 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.959+0000 7f2bba650700 1 --2- 172.21.15.195:0/3079590899 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2bb40f5410 0x7f2bb4105630 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:34.964 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.959+0000 7f2bba650700 1 -- 172.21.15.195:0/3079590899 >> 172.21.15.195:0/3079590899 conn(0x7f2bb40f0b70 msgr2=0x7f2bb40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:34.965 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.960+0000 7f2bba650700 1 -- 172.21.15.195:0/3079590899 shutdown_connections 2023-10-12T22:32:34.965 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:34.960+0000 7f2bba650700 1 -- 172.21.15.195:0/3079590899 wait complete. 2023-10-12T22:32:34.965 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:36.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:35 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:34.805957+0000 mgr.smithi131.efktma (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:36.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:35 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:34.959019+0000 mon.smithi131 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.195:0/3079590899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:36.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:36.365 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:37.471 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.469+0000 7f70e693b700 1 -- 172.21.15.195:0/1890773547 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f70e00f6510 msgr2=0x7f70e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:37.472 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.469+0000 7f70e693b700 1 --2- 172.21.15.195:0/1890773547 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f70e00f6510 0x7f70e00f6930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f70d0009fb0 tx=0x7f70d0009b30).stop 2023-10-12T22:32:37.472 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.469+0000 7f70e693b700 1 -- 172.21.15.195:0/1890773547 shutdown_connections 2023-10-12T22:32:37.472 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.469+0000 7f70e693b700 1 --2- 172.21.15.195:0/1890773547 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f70e00f6510 0x7f70e00f6930 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:37.472 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.469+0000 7f70e693b700 1 -- 172.21.15.195:0/1890773547 >> 172.21.15.195:0/1890773547 conn(0x7f70e00f1c70 msgr2=0x7f70e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:37.472 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.469+0000 7f70e693b700 1 -- 172.21.15.195:0/1890773547 shutdown_connections 2023-10-12T22:32:37.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.469+0000 7f70e693b700 1 -- 172.21.15.195:0/1890773547 wait complete. 2023-10-12T22:32:37.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.470+0000 7f70e693b700 1 Processor -- start 2023-10-12T22:32:37.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.470+0000 7f70e693b700 1 -- start start 2023-10-12T22:32:37.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.470+0000 7f70e693b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f70e00f6510 0x7f70e0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:37.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.470+0000 7f70e693b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e0106c20 con 0x7f70e00f6510 2023-10-12T22:32:37.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.471+0000 7f70dffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f70e00f6510 0x7f70e0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:37.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.471+0000 7f70dffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f70e00f6510 0x7f70e0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38314/0 (socket says 172.21.15.195:38314) 2023-10-12T22:32:37.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.471+0000 7f70dffff700 1 -- 172.21.15.195:0/1337297298 learned_addr learned my addr 172.21.15.195:0/1337297298 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:37.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.471+0000 7f70dffff700 1 -- 172.21.15.195:0/1337297298 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d0005040 con 0x7f70e00f6510 2023-10-12T22:32:37.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.471+0000 7f70dffff700 1 --2- 172.21.15.195:0/1337297298 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f70e00f6510 0x7f70e0106730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f70d0013510 tx=0x7f70d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:37.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.471+0000 7f70dd7fa700 1 -- 172.21.15.195:0/1337297298 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70d0009320 con 0x7f70e00f6510 2023-10-12T22:32:37.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.471+0000 7f70e693b700 1 -- 172.21.15.195:0/1337297298 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e0106e20 con 0x7f70e00f6510 2023-10-12T22:32:37.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.471+0000 7f70dd7fa700 1 -- 172.21.15.195:0/1337297298 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70d0013f00 con 0x7f70e00f6510 2023-10-12T22:32:37.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.471+0000 7f70dd7fa700 1 -- 172.21.15.195:0/1337297298 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70d0014d90 con 0x7f70e00f6510 2023-10-12T22:32:37.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.471+0000 7f70e693b700 1 -- 172.21.15.195:0/1337297298 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e01072b0 con 0x7f70e00f6510 2023-10-12T22:32:37.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.472+0000 7f70dd7fa700 1 -- 172.21.15.195:0/1337297298 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70d0014420 con 0x7f70e00f6510 2023-10-12T22:32:37.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.472+0000 7f70dd7fa700 1 --2- 172.21.15.195:0/1337297298 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f70cc02ffb0 0x7f70cc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:37.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.472+0000 7f70e693b700 1 -- 172.21.15.195:0/1337297298 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e0045ad0 con 0x7f70e00f6510 2023-10-12T22:32:37.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.472+0000 7f70dd7fa700 1 -- 172.21.15.195:0/1337297298 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f70d00410c0 con 0x7f70e00f6510 2023-10-12T22:32:37.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.472+0000 7f70d7fff700 1 --2- 172.21.15.195:0/1337297298 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f70cc02ffb0 0x7f70cc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:37.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.473+0000 7f70d7fff700 1 --2- 172.21.15.195:0/1337297298 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f70cc02ffb0 0x7f70cc032470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f70c80096a0 tx=0x7f70c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:37.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.478+0000 7f70dd7fa700 1 -- 172.21.15.195:0/1337297298 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f70d00095f0 con 0x7f70e00f6510 2023-10-12T22:32:37.682 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.682+0000 7f70e693b700 1 -- 172.21.15.195:0/1337297298 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70e00fc050 con 0x7f70e00f6510 2023-10-12T22:32:37.683 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.683+0000 7f70dd7fa700 1 -- 172.21.15.195:0/1337297298 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f70d002b020 con 0x7f70e00f6510 2023-10-12T22:32:37.684 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:37.684 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:37.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.685+0000 7f70e693b700 1 -- 172.21.15.195:0/1337297298 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f70cc02ffb0 msgr2=0x7f70cc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:37.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.685+0000 7f70e693b700 1 --2- 172.21.15.195:0/1337297298 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f70cc02ffb0 0x7f70cc032470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f70c80096a0 tx=0x7f70c8006b40).stop 2023-10-12T22:32:37.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.685+0000 7f70e693b700 1 -- 172.21.15.195:0/1337297298 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f70e00f6510 msgr2=0x7f70e0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:37.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.685+0000 7f70e693b700 1 --2- 172.21.15.195:0/1337297298 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f70e00f6510 0x7f70e0106730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f70d0013510 tx=0x7f70d000b040).stop 2023-10-12T22:32:37.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.685+0000 7f70e693b700 1 -- 172.21.15.195:0/1337297298 shutdown_connections 2023-10-12T22:32:37.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.685+0000 7f70e693b700 1 --2- 172.21.15.195:0/1337297298 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f70cc02ffb0 0x7f70cc032470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:37.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.685+0000 7f70e693b700 1 --2- 172.21.15.195:0/1337297298 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f70e00f6510 0x7f70e0106730 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:37.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.685+0000 7f70e693b700 1 -- 172.21.15.195:0/1337297298 >> 172.21.15.195:0/1337297298 conn(0x7f70e00f1c70 msgr2=0x7f70e00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:37.687 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.685+0000 7f70e693b700 1 -- 172.21.15.195:0/1337297298 shutdown_connections 2023-10-12T22:32:37.687 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:37.685+0000 7f70e693b700 1 -- 172.21.15.195:0/1337297298 wait complete. 2023-10-12T22:32:37.687 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:38.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:37 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:36.806228+0000 mgr.smithi131.efktma (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:38.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:37 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:37.683349+0000 mon.smithi131 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.195:0/1337297298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:39.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:39.092 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:40.219 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.217+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/2051967500 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3b00f4740 msgr2=0x7ff3b00f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:40.219 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.217+0000 7ff3b4aa2700 1 --2- 172.21.15.195:0/2051967500 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3b00f4740 0x7ff3b00f4b60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7ff3a0009fb0 tx=0x7ff3a0009b30).stop 2023-10-12T22:32:40.219 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.217+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/2051967500 shutdown_connections 2023-10-12T22:32:40.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.217+0000 7ff3b4aa2700 1 --2- 172.21.15.195:0/2051967500 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3b00f4740 0x7ff3b00f4b60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:40.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.217+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/2051967500 >> 172.21.15.195:0/2051967500 conn(0x7ff3b00f0260 msgr2=0x7ff3b00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:40.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.217+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/2051967500 shutdown_connections 2023-10-12T22:32:40.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.217+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/2051967500 wait complete. 2023-10-12T22:32:40.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.218+0000 7ff3b4aa2700 1 Processor -- start 2023-10-12T22:32:40.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.218+0000 7ff3b4aa2700 1 -- start start 2023-10-12T22:32:40.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.218+0000 7ff3b4aa2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3b00f4740 0x7ff3b01066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:40.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.218+0000 7ff3b4aa2700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b0106bc0 con 0x7ff3b00f4740 2023-10-12T22:32:40.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.218+0000 7ff3ae59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3b00f4740 0x7ff3b01066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:40.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.218+0000 7ff3ae59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3b00f4740 0x7ff3b01066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:42556/0 (socket says 172.21.15.195:42556) 2023-10-12T22:32:40.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.218+0000 7ff3ae59c700 1 -- 172.21.15.195:0/1368284275 learned_addr learned my addr 172.21.15.195:0/1368284275 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:40.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.218+0000 7ff3ae59c700 1 -- 172.21.15.195:0/1368284275 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a0005040 con 0x7ff3b00f4740 2023-10-12T22:32:40.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.219+0000 7ff3ae59c700 1 --2- 172.21.15.195:0/1368284275 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3b00f4740 0x7ff3b01066d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7ff3a0013510 tx=0x7ff3a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:40.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.219+0000 7ff39effd700 1 -- 172.21.15.195:0/1368284275 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3a0009320 con 0x7ff3b00f4740 2023-10-12T22:32:40.223 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.219+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/1368284275 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b0106dc0 con 0x7ff3b00f4740 2023-10-12T22:32:40.223 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.219+0000 7ff39effd700 1 -- 172.21.15.195:0/1368284275 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3a0013f00 con 0x7ff3b00f4740 2023-10-12T22:32:40.223 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.219+0000 7ff39effd700 1 -- 172.21.15.195:0/1368284275 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3a0014d90 con 0x7ff3b00f4740 2023-10-12T22:32:40.223 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.219+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/1368284275 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b0107250 con 0x7ff3b00f4740 2023-10-12T22:32:40.225 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.219+0000 7ff39effd700 1 -- 172.21.15.195:0/1368284275 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3a00263a0 con 0x7ff3b00f4740 2023-10-12T22:32:40.225 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.220+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/1368284275 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b0045ad0 con 0x7ff3b00f4740 2023-10-12T22:32:40.225 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.220+0000 7ff39effd700 1 --2- 172.21.15.195:0/1368284275 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff39802ff60 0x7ff398032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:40.225 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.220+0000 7ff39effd700 1 -- 172.21.15.195:0/1368284275 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3a00413b0 con 0x7ff3b00f4740 2023-10-12T22:32:40.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.220+0000 7ff3add9b700 1 --2- 172.21.15.195:0/1368284275 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff39802ff60 0x7ff398032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:40.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.220+0000 7ff3add9b700 1 --2- 172.21.15.195:0/1368284275 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff39802ff60 0x7ff398032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7ff3a40096a0 tx=0x7ff3a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:40.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.224+0000 7ff39effd700 1 -- 172.21.15.195:0/1368284275 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3a00126f0 con 0x7ff3b00f4740 2023-10-12T22:32:40.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:39 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:38.806523+0000 mgr.smithi131.efktma (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:40.435 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.435+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/1368284275 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3b0108100 con 0x7ff3b00f4740 2023-10-12T22:32:40.436 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.436+0000 7ff39effd700 1 -- 172.21.15.195:0/1368284275 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff3a0012330 con 0x7ff3b00f4740 2023-10-12T22:32:40.436 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:40.436 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:40.438 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.437+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/1368284275 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff39802ff60 msgr2=0x7ff398032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:40.438 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.437+0000 7ff3b4aa2700 1 --2- 172.21.15.195:0/1368284275 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff39802ff60 0x7ff398032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7ff3a40096a0 tx=0x7ff3a4006b40).stop 2023-10-12T22:32:40.438 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.437+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/1368284275 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3b00f4740 msgr2=0x7ff3b01066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:40.438 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.437+0000 7ff3b4aa2700 1 --2- 172.21.15.195:0/1368284275 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3b00f4740 0x7ff3b01066d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7ff3a0013510 tx=0x7ff3a000b040).stop 2023-10-12T22:32:40.438 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.437+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/1368284275 shutdown_connections 2023-10-12T22:32:40.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.437+0000 7ff3b4aa2700 1 --2- 172.21.15.195:0/1368284275 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff39802ff60 0x7ff398032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:40.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.437+0000 7ff3b4aa2700 1 --2- 172.21.15.195:0/1368284275 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3b00f4740 0x7ff3b01066d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:40.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.437+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/1368284275 >> 172.21.15.195:0/1368284275 conn(0x7ff3b00f0260 msgr2=0x7ff3b00f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:40.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.437+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/1368284275 shutdown_connections 2023-10-12T22:32:40.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:40.437+0000 7ff3b4aa2700 1 -- 172.21.15.195:0/1368284275 wait complete. 2023-10-12T22:32:40.440 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:41.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:40 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:40.435882+0000 mon.smithi131 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.195:0/1368284275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:41.842 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:41.843 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:42.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:40.806853+0000 mgr.smithi131.efktma (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:42.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.950+0000 7facec6a2700 1 -- 172.21.15.195:0/3025340319 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7face40f6510 msgr2=0x7face40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:42.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.950+0000 7facec6a2700 1 --2- 172.21.15.195:0/3025340319 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7face40f6510 0x7face40f6930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7face0009fb0 tx=0x7face0009b30).stop 2023-10-12T22:32:42.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.951+0000 7facec6a2700 1 -- 172.21.15.195:0/3025340319 shutdown_connections 2023-10-12T22:32:42.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.951+0000 7facec6a2700 1 --2- 172.21.15.195:0/3025340319 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7face40f6510 0x7face40f6930 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:42.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.951+0000 7facec6a2700 1 -- 172.21.15.195:0/3025340319 >> 172.21.15.195:0/3025340319 conn(0x7face40f1c70 msgr2=0x7face40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:42.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.951+0000 7facec6a2700 1 -- 172.21.15.195:0/3025340319 shutdown_connections 2023-10-12T22:32:42.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.951+0000 7facec6a2700 1 -- 172.21.15.195:0/3025340319 wait complete. 2023-10-12T22:32:42.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.951+0000 7facec6a2700 1 Processor -- start 2023-10-12T22:32:42.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.951+0000 7facec6a2700 1 -- start start 2023-10-12T22:32:42.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.951+0000 7facec6a2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7face40f6510 0x7face4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:42.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.951+0000 7facec6a2700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face4106c20 con 0x7face40f6510 2023-10-12T22:32:42.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.952+0000 7facea43e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7face40f6510 0x7face4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:42.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.952+0000 7facea43e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7face40f6510 0x7face4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:42574/0 (socket says 172.21.15.195:42574) 2023-10-12T22:32:42.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.952+0000 7facea43e700 1 -- 172.21.15.195:0/831031187 learned_addr learned my addr 172.21.15.195:0/831031187 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:42.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.952+0000 7facea43e700 1 -- 172.21.15.195:0/831031187 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face0005040 con 0x7face40f6510 2023-10-12T22:32:42.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.952+0000 7facea43e700 1 --2- 172.21.15.195:0/831031187 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7face40f6510 0x7face4106730 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7face0014000 tx=0x7face000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:42.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.952+0000 7facdaffd700 1 -- 172.21.15.195:0/831031187 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7face00095f0 con 0x7face40f6510 2023-10-12T22:32:42.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.952+0000 7facdaffd700 1 -- 172.21.15.195:0/831031187 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7face0007e70 con 0x7face40f6510 2023-10-12T22:32:42.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.952+0000 7facec6a2700 1 -- 172.21.15.195:0/831031187 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face4106e20 con 0x7face40f6510 2023-10-12T22:32:42.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.952+0000 7facdaffd700 1 -- 172.21.15.195:0/831031187 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7face0014ea0 con 0x7face40f6510 2023-10-12T22:32:42.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.952+0000 7facec6a2700 1 -- 172.21.15.195:0/831031187 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face41072b0 con 0x7face40f6510 2023-10-12T22:32:42.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.953+0000 7facdaffd700 1 -- 172.21.15.195:0/831031187 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7face0025070 con 0x7face40f6510 2023-10-12T22:32:42.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.953+0000 7facdaffd700 1 --2- 172.21.15.195:0/831031187 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7facd002ff60 0x7facd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:42.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.953+0000 7facdaffd700 1 -- 172.21.15.195:0/831031187 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7face0040980 con 0x7face40f6510 2023-10-12T22:32:42.958 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.953+0000 7facec6a2700 1 -- 172.21.15.195:0/831031187 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face4045ad0 con 0x7face40f6510 2023-10-12T22:32:42.958 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.954+0000 7face9c3d700 1 --2- 172.21.15.195:0/831031187 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7facd002ff60 0x7facd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:42.958 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.954+0000 7face9c3d700 1 --2- 172.21.15.195:0/831031187 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7facd002ff60 0x7facd0032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7facd40096a0 tx=0x7facd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:42.958 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:42.958+0000 7facdaffd700 1 -- 172.21.15.195:0/831031187 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7face0026350 con 0x7face40f6510 2023-10-12T22:32:43.165 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:43.165+0000 7facec6a2700 1 -- 172.21.15.195:0/831031187 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7face40fc050 con 0x7face40f6510 2023-10-12T22:32:43.166 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:43.165+0000 7facdaffd700 1 -- 172.21.15.195:0/831031187 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7face0012330 con 0x7face40f6510 2023-10-12T22:32:43.166 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:43.167 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:43.168 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:43.167+0000 7facec6a2700 1 -- 172.21.15.195:0/831031187 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7facd002ff60 msgr2=0x7facd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:43.168 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:43.167+0000 7facec6a2700 1 --2- 172.21.15.195:0/831031187 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7facd002ff60 0x7facd0032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7facd40096a0 tx=0x7facd4006b40).stop 2023-10-12T22:32:43.168 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:43.167+0000 7facec6a2700 1 -- 172.21.15.195:0/831031187 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7face40f6510 msgr2=0x7face4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:43.168 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:43.167+0000 7facec6a2700 1 --2- 172.21.15.195:0/831031187 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7face40f6510 0x7face4106730 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7face0014000 tx=0x7face000b040).stop 2023-10-12T22:32:43.169 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:43.167+0000 7facec6a2700 1 -- 172.21.15.195:0/831031187 shutdown_connections 2023-10-12T22:32:43.169 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:43.167+0000 7facec6a2700 1 --2- 172.21.15.195:0/831031187 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7facd002ff60 0x7facd0032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:43.169 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:43.167+0000 7facec6a2700 1 --2- 172.21.15.195:0/831031187 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7face40f6510 0x7face4106730 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:43.169 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:43.167+0000 7facec6a2700 1 -- 172.21.15.195:0/831031187 >> 172.21.15.195:0/831031187 conn(0x7face40f1c70 msgr2=0x7face40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:43.169 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:43.167+0000 7facec6a2700 1 -- 172.21.15.195:0/831031187 shutdown_connections 2023-10-12T22:32:43.170 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:43.167+0000 7facec6a2700 1 -- 172.21.15.195:0/831031187 wait complete. 2023-10-12T22:32:43.170 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:44.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:43 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:42.807168+0000 mgr.smithi131.efktma (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:44.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:43 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:43.165902+0000 mon.smithi131 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.195:0/831031187' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:44.576 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:44.577 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:45.630 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.629+0000 7f37e3112700 1 -- 172.21.15.195:0/593362160 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37dc060280 msgr2=0x7f37dc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:45.630 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.629+0000 7f37e3112700 1 --2- 172.21.15.195:0/593362160 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37dc060280 0x7f37dc0f90f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f37d0009fb0 tx=0x7f37d0009b30).stop 2023-10-12T22:32:45.631 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.629+0000 7f37e3112700 1 -- 172.21.15.195:0/593362160 shutdown_connections 2023-10-12T22:32:45.631 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.629+0000 7f37e3112700 1 --2- 172.21.15.195:0/593362160 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37dc060280 0x7f37dc0f90f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:45.631 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.629+0000 7f37e3112700 1 -- 172.21.15.195:0/593362160 >> 172.21.15.195:0/593362160 conn(0x7f37dc0f0b70 msgr2=0x7f37dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:45.631 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.629+0000 7f37e3112700 1 -- 172.21.15.195:0/593362160 shutdown_connections 2023-10-12T22:32:45.631 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.629+0000 7f37e3112700 1 -- 172.21.15.195:0/593362160 wait complete. 2023-10-12T22:32:45.632 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.629+0000 7f37e3112700 1 Processor -- start 2023-10-12T22:32:45.632 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.630+0000 7f37e3112700 1 -- start start 2023-10-12T22:32:45.632 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.630+0000 7f37e3112700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37dc060280 0x7f37dc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:45.632 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.630+0000 7f37e3112700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc106bd0 con 0x7f37dc060280 2023-10-12T22:32:45.632 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.630+0000 7f37e0eae700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37dc060280 0x7f37dc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:45.633 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.630+0000 7f37e0eae700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37dc060280 0x7f37dc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:42596/0 (socket says 172.21.15.195:42596) 2023-10-12T22:32:45.633 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.630+0000 7f37e0eae700 1 -- 172.21.15.195:0/3331840045 learned_addr learned my addr 172.21.15.195:0/3331840045 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:45.635 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.630+0000 7f37e0eae700 1 -- 172.21.15.195:0/3331840045 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d0005040 con 0x7f37dc060280 2023-10-12T22:32:45.635 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.631+0000 7f37e0eae700 1 --2- 172.21.15.195:0/3331840045 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37dc060280 0x7f37dc1066e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f37d0013510 tx=0x7f37d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:45.635 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.631+0000 7f37d97fa700 1 -- 172.21.15.195:0/3331840045 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37d0009320 con 0x7f37dc060280 2023-10-12T22:32:45.636 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.631+0000 7f37e3112700 1 -- 172.21.15.195:0/3331840045 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37dc106dd0 con 0x7f37dc060280 2023-10-12T22:32:45.636 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.631+0000 7f37e3112700 1 -- 172.21.15.195:0/3331840045 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37dc107260 con 0x7f37dc060280 2023-10-12T22:32:45.636 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.631+0000 7f37d97fa700 1 -- 172.21.15.195:0/3331840045 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37d0013f00 con 0x7f37dc060280 2023-10-12T22:32:45.636 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.631+0000 7f37d97fa700 1 -- 172.21.15.195:0/3331840045 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37d0014d90 con 0x7f37dc060280 2023-10-12T22:32:45.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.631+0000 7f37d97fa700 1 -- 172.21.15.195:0/3331840045 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37d00263a0 con 0x7f37dc060280 2023-10-12T22:32:45.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.632+0000 7f37e3112700 1 -- 172.21.15.195:0/3331840045 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37dc045ad0 con 0x7f37dc060280 2023-10-12T22:32:45.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.632+0000 7f37d97fa700 1 --2- 172.21.15.195:0/3331840045 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f37cc02ff60 0x7f37cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:45.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.632+0000 7f37d97fa700 1 -- 172.21.15.195:0/3331840045 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37d0041400 con 0x7f37dc060280 2023-10-12T22:32:45.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.632+0000 7f37dbfff700 1 --2- 172.21.15.195:0/3331840045 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f37cc02ff60 0x7f37cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:45.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.632+0000 7f37dbfff700 1 --2- 172.21.15.195:0/3331840045 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f37cc02ff60 0x7f37cc032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f37c80096a0 tx=0x7f37c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:45.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.636+0000 7f37d97fa700 1 -- 172.21.15.195:0/3331840045 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37d0012740 con 0x7f37dc060280 2023-10-12T22:32:45.845 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.845+0000 7f37e3112700 1 -- 172.21.15.195:0/3331840045 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37dc108110 con 0x7f37dc060280 2023-10-12T22:32:45.846 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.846+0000 7f37d97fa700 1 -- 172.21.15.195:0/3331840045 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f37d0012330 con 0x7f37dc060280 2023-10-12T22:32:45.846 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:45.846 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:45.848 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.847+0000 7f37e3112700 1 -- 172.21.15.195:0/3331840045 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f37cc02ff60 msgr2=0x7f37cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:45.848 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.847+0000 7f37e3112700 1 --2- 172.21.15.195:0/3331840045 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f37cc02ff60 0x7f37cc032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f37c80096a0 tx=0x7f37c8006b40).stop 2023-10-12T22:32:45.848 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.847+0000 7f37e3112700 1 -- 172.21.15.195:0/3331840045 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37dc060280 msgr2=0x7f37dc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:45.848 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.847+0000 7f37e3112700 1 --2- 172.21.15.195:0/3331840045 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37dc060280 0x7f37dc1066e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f37d0013510 tx=0x7f37d000b040).stop 2023-10-12T22:32:45.849 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.848+0000 7f37e3112700 1 -- 172.21.15.195:0/3331840045 shutdown_connections 2023-10-12T22:32:45.849 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.848+0000 7f37e3112700 1 --2- 172.21.15.195:0/3331840045 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f37cc02ff60 0x7f37cc032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:45.849 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.848+0000 7f37e3112700 1 --2- 172.21.15.195:0/3331840045 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37dc060280 0x7f37dc1066e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:45.849 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.848+0000 7f37e3112700 1 -- 172.21.15.195:0/3331840045 >> 172.21.15.195:0/3331840045 conn(0x7f37dc0f0b70 msgr2=0x7f37dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:45.849 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.848+0000 7f37e3112700 1 -- 172.21.15.195:0/3331840045 shutdown_connections 2023-10-12T22:32:45.849 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:45.848+0000 7f37e3112700 1 -- 172.21.15.195:0/3331840045 wait complete. 2023-10-12T22:32:45.850 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:46.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:45 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:44.807474+0000 mgr.smithi131.efktma (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:46.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:45 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:45.846071+0000 mon.smithi131 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.195:0/3331840045' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:47.265 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:47.266 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:48.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:47 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:46.807789+0000 mgr.smithi131.efktma (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:48.400 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.398+0000 7f78bd0c7700 1 -- 172.21.15.195:0/1627788104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78b80f6510 msgr2=0x7f78b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:48.400 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.398+0000 7f78bd0c7700 1 --2- 172.21.15.195:0/1627788104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78b80f6510 0x7f78b80f6930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f78ac009fb0 tx=0x7f78ac009b30).stop 2023-10-12T22:32:48.400 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.398+0000 7f78bd0c7700 1 -- 172.21.15.195:0/1627788104 shutdown_connections 2023-10-12T22:32:48.400 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.398+0000 7f78bd0c7700 1 --2- 172.21.15.195:0/1627788104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78b80f6510 0x7f78b80f6930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:48.401 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.398+0000 7f78bd0c7700 1 -- 172.21.15.195:0/1627788104 >> 172.21.15.195:0/1627788104 conn(0x7f78b80f1c70 msgr2=0x7f78b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:48.401 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.398+0000 7f78bd0c7700 1 -- 172.21.15.195:0/1627788104 shutdown_connections 2023-10-12T22:32:48.401 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.398+0000 7f78bd0c7700 1 -- 172.21.15.195:0/1627788104 wait complete. 2023-10-12T22:32:48.401 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.399+0000 7f78bd0c7700 1 Processor -- start 2023-10-12T22:32:48.401 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.399+0000 7f78bd0c7700 1 -- start start 2023-10-12T22:32:48.402 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.399+0000 7f78bd0c7700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78b80f6510 0x7f78b8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:48.402 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.399+0000 7f78bd0c7700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b8106c20 con 0x7f78b80f6510 2023-10-12T22:32:48.402 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.399+0000 7f78b6d9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78b80f6510 0x7f78b8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:48.402 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.399+0000 7f78b6d9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78b80f6510 0x7f78b8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:54362/0 (socket says 172.21.15.195:54362) 2023-10-12T22:32:48.402 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.399+0000 7f78b6d9d700 1 -- 172.21.15.195:0/3170767589 learned_addr learned my addr 172.21.15.195:0/3170767589 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:48.403 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.400+0000 7f78b6d9d700 1 -- 172.21.15.195:0/3170767589 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac005040 con 0x7f78b80f6510 2023-10-12T22:32:48.403 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.400+0000 7f78b6d9d700 1 --2- 172.21.15.195:0/3170767589 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78b80f6510 0x7f78b8106730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f78ac013510 tx=0x7f78ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:48.404 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.400+0000 7f789f7fe700 1 -- 172.21.15.195:0/3170767589 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78ac009320 con 0x7f78b80f6510 2023-10-12T22:32:48.404 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.400+0000 7f78bd0c7700 1 -- 172.21.15.195:0/3170767589 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b8106e20 con 0x7f78b80f6510 2023-10-12T22:32:48.404 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.400+0000 7f789f7fe700 1 -- 172.21.15.195:0/3170767589 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78ac013f00 con 0x7f78b80f6510 2023-10-12T22:32:48.405 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.400+0000 7f789f7fe700 1 -- 172.21.15.195:0/3170767589 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78ac014d90 con 0x7f78b80f6510 2023-10-12T22:32:48.405 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.400+0000 7f78bd0c7700 1 -- 172.21.15.195:0/3170767589 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b81072b0 con 0x7f78b80f6510 2023-10-12T22:32:48.405 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.401+0000 7f789f7fe700 1 -- 172.21.15.195:0/3170767589 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78ac014420 con 0x7f78b80f6510 2023-10-12T22:32:48.405 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.401+0000 7f78bd0c7700 1 -- 172.21.15.195:0/3170767589 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b8045ad0 con 0x7f78b80f6510 2023-10-12T22:32:48.405 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.401+0000 7f789f7fe700 1 --2- 172.21.15.195:0/3170767589 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f78a002ff60 0x7f78a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:48.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.401+0000 7f789f7fe700 1 -- 172.21.15.195:0/3170767589 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78ac041110 con 0x7f78b80f6510 2023-10-12T22:32:48.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.401+0000 7f78b659c700 1 --2- 172.21.15.195:0/3170767589 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f78a002ff60 0x7f78a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:48.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.401+0000 7f78b659c700 1 --2- 172.21.15.195:0/3170767589 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f78a002ff60 0x7f78a0032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f78a80096a0 tx=0x7f78a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:48.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.405+0000 7f789f7fe700 1 -- 172.21.15.195:0/3170767589 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78ac0095f0 con 0x7f78b80f6510 2023-10-12T22:32:48.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.613+0000 7f78bd0c7700 1 -- 172.21.15.195:0/3170767589 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78b80fbf60 con 0x7f78b80f6510 2023-10-12T22:32:48.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.613+0000 7f789f7fe700 1 -- 172.21.15.195:0/3170767589 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f78ac01db70 con 0x7f78b80f6510 2023-10-12T22:32:48.614 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:48.614 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:48.615 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.615+0000 7f78bd0c7700 1 -- 172.21.15.195:0/3170767589 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f78a002ff60 msgr2=0x7f78a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:48.616 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.615+0000 7f78bd0c7700 1 --2- 172.21.15.195:0/3170767589 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f78a002ff60 0x7f78a0032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f78a80096a0 tx=0x7f78a8006b40).stop 2023-10-12T22:32:48.616 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.615+0000 7f78bd0c7700 1 -- 172.21.15.195:0/3170767589 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78b80f6510 msgr2=0x7f78b8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:48.616 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.615+0000 7f78bd0c7700 1 --2- 172.21.15.195:0/3170767589 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78b80f6510 0x7f78b8106730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f78ac013510 tx=0x7f78ac00b040).stop 2023-10-12T22:32:48.616 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.615+0000 7f78bd0c7700 1 -- 172.21.15.195:0/3170767589 shutdown_connections 2023-10-12T22:32:48.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.615+0000 7f78bd0c7700 1 --2- 172.21.15.195:0/3170767589 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f78a002ff60 0x7f78a0032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:48.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.615+0000 7f78bd0c7700 1 --2- 172.21.15.195:0/3170767589 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78b80f6510 0x7f78b8106730 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:48.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.615+0000 7f78bd0c7700 1 -- 172.21.15.195:0/3170767589 >> 172.21.15.195:0/3170767589 conn(0x7f78b80f1c70 msgr2=0x7f78b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:48.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.615+0000 7f78bd0c7700 1 -- 172.21.15.195:0/3170767589 shutdown_connections 2023-10-12T22:32:48.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:48.615+0000 7f78bd0c7700 1 -- 172.21.15.195:0/3170767589 wait complete. 2023-10-12T22:32:48.618 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:49.242 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:48 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:48.613799+0000 mon.smithi131 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.195:0/3170767589' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:50.030 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:50.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:50.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:49 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:48.808121+0000 mgr.smithi131.efktma (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:51.190 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.189+0000 7f54cd16f700 1 -- 172.21.15.195:0/4174619623 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c80f4740 msgr2=0x7f54c80f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.189+0000 7f54cd16f700 1 --2- 172.21.15.195:0/4174619623 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c80f4740 0x7f54c80f4b60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f54b8009fb0 tx=0x7f54b8009b30).stop 2023-10-12T22:32:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.189+0000 7f54cd16f700 1 -- 172.21.15.195:0/4174619623 shutdown_connections 2023-10-12T22:32:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.189+0000 7f54cd16f700 1 --2- 172.21.15.195:0/4174619623 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c80f4740 0x7f54c80f4b60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.189+0000 7f54cd16f700 1 -- 172.21.15.195:0/4174619623 >> 172.21.15.195:0/4174619623 conn(0x7f54c80f0260 msgr2=0x7f54c80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.189+0000 7f54cd16f700 1 -- 172.21.15.195:0/4174619623 shutdown_connections 2023-10-12T22:32:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.189+0000 7f54cd16f700 1 -- 172.21.15.195:0/4174619623 wait complete. 2023-10-12T22:32:51.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.190+0000 7f54cd16f700 1 Processor -- start 2023-10-12T22:32:51.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.190+0000 7f54cd16f700 1 -- start start 2023-10-12T22:32:51.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.190+0000 7f54cd16f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c80f4740 0x7f54c81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:51.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.190+0000 7f54cd16f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c8106bc0 con 0x7f54c80f4740 2023-10-12T22:32:51.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.190+0000 7f54c6d9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c80f4740 0x7f54c81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:51.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.190+0000 7f54c6d9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c80f4740 0x7f54c81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:54382/0 (socket says 172.21.15.195:54382) 2023-10-12T22:32:51.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.190+0000 7f54c6d9d700 1 -- 172.21.15.195:0/4224024319 learned_addr learned my addr 172.21.15.195:0/4224024319 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:51.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.191+0000 7f54c6d9d700 1 -- 172.21.15.195:0/4224024319 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b8005040 con 0x7f54c80f4740 2023-10-12T22:32:51.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.191+0000 7f54c6d9d700 1 --2- 172.21.15.195:0/4224024319 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c80f4740 0x7f54c81066d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f54b8013510 tx=0x7f54b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:51.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.191+0000 7f54bf7fe700 1 -- 172.21.15.195:0/4224024319 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54b8009320 con 0x7f54c80f4740 2023-10-12T22:32:51.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.191+0000 7f54cd16f700 1 -- 172.21.15.195:0/4224024319 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c8106dc0 con 0x7f54c80f4740 2023-10-12T22:32:51.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.191+0000 7f54bf7fe700 1 -- 172.21.15.195:0/4224024319 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54b8013f00 con 0x7f54c80f4740 2023-10-12T22:32:51.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.191+0000 7f54bf7fe700 1 -- 172.21.15.195:0/4224024319 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54b8014d90 con 0x7f54c80f4740 2023-10-12T22:32:51.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.191+0000 7f54cd16f700 1 -- 172.21.15.195:0/4224024319 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c8107250 con 0x7f54c80f4740 2023-10-12T22:32:51.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.192+0000 7f54bf7fe700 1 -- 172.21.15.195:0/4224024319 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54b80095f0 con 0x7f54c80f4740 2023-10-12T22:32:51.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.192+0000 7f54cd16f700 1 -- 172.21.15.195:0/4224024319 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c8045ad0 con 0x7f54c80f4740 2023-10-12T22:32:51.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.192+0000 7f54bf7fe700 1 --2- 172.21.15.195:0/4224024319 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f54b402ffb0 0x7f54b4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:51.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.192+0000 7f54bf7fe700 1 -- 172.21.15.195:0/4224024319 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54b8040f80 con 0x7f54c80f4740 2023-10-12T22:32:51.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.192+0000 7f54c659c700 1 --2- 172.21.15.195:0/4224024319 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f54b402ffb0 0x7f54b4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:51.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.193+0000 7f54c659c700 1 --2- 172.21.15.195:0/4224024319 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f54b402ffb0 0x7f54b4032470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f54b00096a0 tx=0x7f54b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:51.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.197+0000 7f54bf7fe700 1 -- 172.21.15.195:0/4224024319 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54b8014420 con 0x7f54c80f4740 2023-10-12T22:32:51.402 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.401+0000 7f54cd16f700 1 -- 172.21.15.195:0/4224024319 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54c8108100 con 0x7f54c80f4740 2023-10-12T22:32:51.403 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.402+0000 7f54bf7fe700 1 -- 172.21.15.195:0/4224024319 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f54b801d7f0 con 0x7f54c80f4740 2023-10-12T22:32:51.403 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:51.403 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:51.404 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.403+0000 7f54cd16f700 1 -- 172.21.15.195:0/4224024319 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f54b402ffb0 msgr2=0x7f54b4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:51.405 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.403+0000 7f54cd16f700 1 --2- 172.21.15.195:0/4224024319 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f54b402ffb0 0x7f54b4032470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f54b00096a0 tx=0x7f54b0006b40).stop 2023-10-12T22:32:51.405 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.403+0000 7f54cd16f700 1 -- 172.21.15.195:0/4224024319 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c80f4740 msgr2=0x7f54c81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:51.405 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.403+0000 7f54cd16f700 1 --2- 172.21.15.195:0/4224024319 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c80f4740 0x7f54c81066d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f54b8013510 tx=0x7f54b800b040).stop 2023-10-12T22:32:51.405 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.403+0000 7f54cd16f700 1 -- 172.21.15.195:0/4224024319 shutdown_connections 2023-10-12T22:32:51.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.403+0000 7f54cd16f700 1 --2- 172.21.15.195:0/4224024319 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f54b402ffb0 0x7f54b4032470 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:51.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.403+0000 7f54cd16f700 1 --2- 172.21.15.195:0/4224024319 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54c80f4740 0x7f54c81066d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:51.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.403+0000 7f54cd16f700 1 -- 172.21.15.195:0/4224024319 >> 172.21.15.195:0/4224024319 conn(0x7f54c80f0260 msgr2=0x7f54c80f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:51.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.403+0000 7f54cd16f700 1 -- 172.21.15.195:0/4224024319 shutdown_connections 2023-10-12T22:32:51.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:51.403+0000 7f54cd16f700 1 -- 172.21.15.195:0/4224024319 wait complete. 2023-10-12T22:32:51.407 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:52.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:51 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:50.808427+0000 mgr.smithi131.efktma (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:52.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:51 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:51.403011+0000 mon.smithi131 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.195:0/4224024319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:52.738 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:52.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:53.885 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.883+0000 7fd9aee40700 1 -- 172.21.15.195:0/1028794396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a8060280 msgr2=0x7fd9a80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:53.885 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.883+0000 7fd9aee40700 1 --2- 172.21.15.195:0/1028794396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a8060280 0x7fd9a80f90f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fd99c009fb0 tx=0x7fd99c009b30).stop 2023-10-12T22:32:53.885 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.883+0000 7fd9aee40700 1 -- 172.21.15.195:0/1028794396 shutdown_connections 2023-10-12T22:32:53.885 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.883+0000 7fd9aee40700 1 --2- 172.21.15.195:0/1028794396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a8060280 0x7fd9a80f90f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:53.886 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.883+0000 7fd9aee40700 1 -- 172.21.15.195:0/1028794396 >> 172.21.15.195:0/1028794396 conn(0x7fd9a80f0b70 msgr2=0x7fd9a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:53.886 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.883+0000 7fd9aee40700 1 -- 172.21.15.195:0/1028794396 shutdown_connections 2023-10-12T22:32:53.886 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.883+0000 7fd9aee40700 1 -- 172.21.15.195:0/1028794396 wait complete. 2023-10-12T22:32:53.888 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.884+0000 7fd9aee40700 1 Processor -- start 2023-10-12T22:32:53.888 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.884+0000 7fd9aee40700 1 -- start start 2023-10-12T22:32:53.888 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.884+0000 7fd9aee40700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a8060280 0x7fd9a81088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:53.889 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.884+0000 7fd9aee40700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a8108dc0 con 0x7fd9a8060280 2023-10-12T22:32:53.889 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.885+0000 7fd9acbdc700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a8060280 0x7fd9a81088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:53.889 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.885+0000 7fd9acbdc700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a8060280 0x7fd9a81088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:54400/0 (socket says 172.21.15.195:54400) 2023-10-12T22:32:53.889 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.885+0000 7fd9acbdc700 1 -- 172.21.15.195:0/1563249495 learned_addr learned my addr 172.21.15.195:0/1563249495 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:53.890 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.885+0000 7fd9acbdc700 1 -- 172.21.15.195:0/1563249495 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c005040 con 0x7fd9a8060280 2023-10-12T22:32:53.890 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.885+0000 7fd9acbdc700 1 --2- 172.21.15.195:0/1563249495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a8060280 0x7fd9a81088d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fd99c009830 tx=0x7fd99c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:53.890 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.885+0000 7fd9a57fa700 1 -- 172.21.15.195:0/1563249495 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd99c009320 con 0x7fd9a8060280 2023-10-12T22:32:53.890 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.885+0000 7fd9aee40700 1 -- 172.21.15.195:0/1563249495 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a8108fc0 con 0x7fd9a8060280 2023-10-12T22:32:53.890 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.885+0000 7fd9a57fa700 1 -- 172.21.15.195:0/1563249495 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd99c0136c0 con 0x7fd9a8060280 2023-10-12T22:32:53.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.885+0000 7fd9a57fa700 1 -- 172.21.15.195:0/1563249495 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd99c01ee90 con 0x7fd9a8060280 2023-10-12T22:32:53.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.885+0000 7fd9aee40700 1 -- 172.21.15.195:0/1563249495 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a8109450 con 0x7fd9a8060280 2023-10-12T22:32:53.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.886+0000 7fd9a57fa700 1 -- 172.21.15.195:0/1563249495 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd99c01e420 con 0x7fd9a8060280 2023-10-12T22:32:53.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.886+0000 7fd9aee40700 1 -- 172.21.15.195:0/1563249495 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a8045ad0 con 0x7fd9a8060280 2023-10-12T22:32:53.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.886+0000 7fd9a57fa700 1 --2- 172.21.15.195:0/1563249495 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd99802ff10 0x7fd9980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:53.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.886+0000 7fd9a57fa700 1 -- 172.21.15.195:0/1563249495 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd99c041020 con 0x7fd9a8060280 2023-10-12T22:32:53.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.887+0000 7fd9a7fff700 1 --2- 172.21.15.195:0/1563249495 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd99802ff10 0x7fd9980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:53.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.887+0000 7fd9a7fff700 1 --2- 172.21.15.195:0/1563249495 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd99802ff10 0x7fd9980323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fd9940096a0 tx=0x7fd994006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:53.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:53.891+0000 7fd9a57fa700 1 -- 172.21.15.195:0/1563249495 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd99c00e2a0 con 0x7fd9a8060280 2023-10-12T22:32:54.101 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:54.099+0000 7fd9aee40700 1 -- 172.21.15.195:0/1563249495 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9a810a300 con 0x7fd9a8060280 2023-10-12T22:32:54.101 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:54.099+0000 7fd9a57fa700 1 -- 172.21.15.195:0/1563249495 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd99c00e2a0 con 0x7fd9a8060280 2023-10-12T22:32:54.102 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:54.103 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:54.104 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:54.101+0000 7fd9aee40700 1 -- 172.21.15.195:0/1563249495 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd99802ff10 msgr2=0x7fd9980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:54.104 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:54.101+0000 7fd9aee40700 1 --2- 172.21.15.195:0/1563249495 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd99802ff10 0x7fd9980323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fd9940096a0 tx=0x7fd994006b40).stop 2023-10-12T22:32:54.105 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:54.101+0000 7fd9aee40700 1 -- 172.21.15.195:0/1563249495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a8060280 msgr2=0x7fd9a81088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:54.105 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:54.101+0000 7fd9aee40700 1 --2- 172.21.15.195:0/1563249495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a8060280 0x7fd9a81088d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fd99c009830 tx=0x7fd99c00b040).stop 2023-10-12T22:32:54.105 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:54.101+0000 7fd9aee40700 1 -- 172.21.15.195:0/1563249495 shutdown_connections 2023-10-12T22:32:54.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:54.101+0000 7fd9aee40700 1 --2- 172.21.15.195:0/1563249495 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd99802ff10 0x7fd9980323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:54.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:54.101+0000 7fd9aee40700 1 --2- 172.21.15.195:0/1563249495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9a8060280 0x7fd9a81088d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:54.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:54.101+0000 7fd9aee40700 1 -- 172.21.15.195:0/1563249495 >> 172.21.15.195:0/1563249495 conn(0x7fd9a80f0b70 msgr2=0x7fd9a80f15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:54.107 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:54.101+0000 7fd9aee40700 1 -- 172.21.15.195:0/1563249495 shutdown_connections 2023-10-12T22:32:54.107 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:54.101+0000 7fd9aee40700 1 -- 172.21.15.195:0/1563249495 wait complete. 2023-10-12T22:32:54.107 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:54.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:53 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:52.808764+0000 mgr.smithi131.efktma (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:55.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:54.100903+0000 mon.smithi131 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.195:0/1563249495' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:55.495 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:55.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:56.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:55 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:54.809054+0000 mgr.smithi131.efktma (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:56.706 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.704+0000 7f604665d700 1 -- 172.21.15.195:0/926037248 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60400f6480 msgr2=0x7f60400f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:56.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.704+0000 7f604665d700 1 --2- 172.21.15.195:0/926037248 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60400f6480 0x7f60400f68a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f6030009fb0 tx=0x7f6030009b30).stop 2023-10-12T22:32:56.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.704+0000 7f604665d700 1 -- 172.21.15.195:0/926037248 shutdown_connections 2023-10-12T22:32:56.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.704+0000 7f604665d700 1 --2- 172.21.15.195:0/926037248 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60400f6480 0x7f60400f68a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:56.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.704+0000 7f604665d700 1 -- 172.21.15.195:0/926037248 >> 172.21.15.195:0/926037248 conn(0x7f60400f1c00 msgr2=0x7f60400f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:56.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.704+0000 7f604665d700 1 -- 172.21.15.195:0/926037248 shutdown_connections 2023-10-12T22:32:56.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.704+0000 7f604665d700 1 -- 172.21.15.195:0/926037248 wait complete. 2023-10-12T22:32:56.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.704+0000 7f604665d700 1 Processor -- start 2023-10-12T22:32:56.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.705+0000 7f604665d700 1 -- start start 2023-10-12T22:32:56.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.705+0000 7f604665d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60400f6480 0x7f60401066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:56.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.705+0000 7f604665d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6040106bc0 con 0x7f60400f6480 2023-10-12T22:32:56.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.705+0000 7f603ffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60400f6480 0x7f60401066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:56.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.705+0000 7f603ffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60400f6480 0x7f60401066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:54410/0 (socket says 172.21.15.195:54410) 2023-10-12T22:32:56.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.705+0000 7f603ffff700 1 -- 172.21.15.195:0/1349545835 learned_addr learned my addr 172.21.15.195:0/1349545835 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:56.711 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.706+0000 7f603ffff700 1 -- 172.21.15.195:0/1349545835 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6030005040 con 0x7f60400f6480 2023-10-12T22:32:56.711 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.706+0000 7f603ffff700 1 --2- 172.21.15.195:0/1349545835 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60400f6480 0x7f60401066d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f6030013510 tx=0x7f603000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:56.711 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.706+0000 7f603cff9700 1 -- 172.21.15.195:0/1349545835 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6030009320 con 0x7f60400f6480 2023-10-12T22:32:56.711 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.706+0000 7f604665d700 1 -- 172.21.15.195:0/1349545835 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6040106dc0 con 0x7f60400f6480 2023-10-12T22:32:56.711 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.706+0000 7f603cff9700 1 -- 172.21.15.195:0/1349545835 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6030013f00 con 0x7f60400f6480 2023-10-12T22:32:56.712 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.706+0000 7f603cff9700 1 -- 172.21.15.195:0/1349545835 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6030014d90 con 0x7f60400f6480 2023-10-12T22:32:56.712 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.706+0000 7f604665d700 1 -- 172.21.15.195:0/1349545835 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6040107250 con 0x7f60400f6480 2023-10-12T22:32:56.712 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.707+0000 7f603cff9700 1 -- 172.21.15.195:0/1349545835 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6030014420 con 0x7f60400f6480 2023-10-12T22:32:56.712 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.707+0000 7f604665d700 1 -- 172.21.15.195:0/1349545835 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6040045ad0 con 0x7f60400f6480 2023-10-12T22:32:56.712 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.707+0000 7f603cff9700 1 --2- 172.21.15.195:0/1349545835 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f602802ff60 0x7f6028032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:56.713 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.707+0000 7f603cff9700 1 -- 172.21.15.195:0/1349545835 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6030040fb0 con 0x7f60400f6480 2023-10-12T22:32:56.713 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.707+0000 7f603f7fe700 1 --2- 172.21.15.195:0/1349545835 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f602802ff60 0x7f6028032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:56.713 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.708+0000 7f603f7fe700 1 --2- 172.21.15.195:0/1349545835 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f602802ff60 0x7f6028032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f60340096a0 tx=0x7f6034006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:56.713 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.712+0000 7f603cff9700 1 -- 172.21.15.195:0/1349545835 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60300095f0 con 0x7f60400f6480 2023-10-12T22:32:56.916 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.915+0000 7f604665d700 1 -- 172.21.15.195:0/1349545835 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6040108100 con 0x7f60400f6480 2023-10-12T22:32:56.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.916+0000 7f603cff9700 1 -- 172.21.15.195:0/1349545835 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f603001db70 con 0x7f60400f6480 2023-10-12T22:32:56.918 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:32:56.918 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:32:56.919 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.918+0000 7f604665d700 1 -- 172.21.15.195:0/1349545835 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f602802ff60 msgr2=0x7f6028032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:56.919 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.918+0000 7f604665d700 1 --2- 172.21.15.195:0/1349545835 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f602802ff60 0x7f6028032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f60340096a0 tx=0x7f6034006b40).stop 2023-10-12T22:32:56.920 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.918+0000 7f604665d700 1 -- 172.21.15.195:0/1349545835 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60400f6480 msgr2=0x7f60401066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:56.920 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.918+0000 7f604665d700 1 --2- 172.21.15.195:0/1349545835 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60400f6480 0x7f60401066d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f6030013510 tx=0x7f603000b040).stop 2023-10-12T22:32:56.920 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.918+0000 7f604665d700 1 -- 172.21.15.195:0/1349545835 shutdown_connections 2023-10-12T22:32:56.920 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.918+0000 7f604665d700 1 --2- 172.21.15.195:0/1349545835 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f602802ff60 0x7f6028032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:56.920 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.918+0000 7f604665d700 1 --2- 172.21.15.195:0/1349545835 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60400f6480 0x7f60401066d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:56.921 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.918+0000 7f604665d700 1 -- 172.21.15.195:0/1349545835 >> 172.21.15.195:0/1349545835 conn(0x7f60400f1c00 msgr2=0x7f60400f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:56.921 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.918+0000 7f604665d700 1 -- 172.21.15.195:0/1349545835 shutdown_connections 2023-10-12T22:32:56.921 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:56.918+0000 7f604665d700 1 -- 172.21.15.195:0/1349545835 wait complete. 2023-10-12T22:32:56.921 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:32:57.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:32:56.917419+0000 mon.smithi131 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.195:0/1349545835' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:32:58.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:57 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:56.809392+0000 mgr.smithi131.efktma (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:32:58.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:32:58.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:32:59.840 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.836+0000 7fbcd4496700 1 -- 172.21.15.195:0/2825291208 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbccc0f5410 msgr2=0x7fbccc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:32:59.840 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.836+0000 7fbcd4496700 1 --2- 172.21.15.195:0/2825291208 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbccc0f5410 0x7fbccc0f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fbcc0009fb0 tx=0x7fbcc0009b30).stop 2023-10-12T22:32:59.840 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.837+0000 7fbcd4496700 1 -- 172.21.15.195:0/2825291208 shutdown_connections 2023-10-12T22:32:59.840 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.837+0000 7fbcd4496700 1 --2- 172.21.15.195:0/2825291208 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbccc0f5410 0x7fbccc0f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:32:59.841 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.837+0000 7fbcd4496700 1 -- 172.21.15.195:0/2825291208 >> 172.21.15.195:0/2825291208 conn(0x7fbccc0f0b70 msgr2=0x7fbccc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:32:59.841 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.837+0000 7fbcd4496700 1 -- 172.21.15.195:0/2825291208 shutdown_connections 2023-10-12T22:32:59.841 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.837+0000 7fbcd4496700 1 -- 172.21.15.195:0/2825291208 wait complete. 2023-10-12T22:32:59.841 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.837+0000 7fbcd4496700 1 Processor -- start 2023-10-12T22:32:59.841 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.838+0000 7fbcd4496700 1 -- start start 2023-10-12T22:32:59.842 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.838+0000 7fbcd4496700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbccc0f5410 0x7fbccc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:59.842 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.838+0000 7fbcd4496700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc060240 con 0x7fbccc0f5410 2023-10-12T22:32:59.842 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.838+0000 7fbcd2232700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbccc0f5410 0x7fbccc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:59.842 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.838+0000 7fbcd2232700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbccc0f5410 0x7fbccc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:35056/0 (socket says 172.21.15.195:35056) 2023-10-12T22:32:59.842 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.838+0000 7fbcd2232700 1 -- 172.21.15.195:0/2537369625 learned_addr learned my addr 172.21.15.195:0/2537369625 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:32:59.843 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.838+0000 7fbcd2232700 1 -- 172.21.15.195:0/2537369625 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc0005040 con 0x7fbccc0f5410 2023-10-12T22:32:59.843 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.838+0000 7fbcd2232700 1 --2- 172.21.15.195:0/2537369625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbccc0f5410 0x7fbccc109430 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fbcc0013510 tx=0x7fbcc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:59.843 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.838+0000 7fbcbeffd700 1 -- 172.21.15.195:0/2537369625 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcc0009320 con 0x7fbccc0f5410 2023-10-12T22:32:59.843 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.839+0000 7fbcd4496700 1 -- 172.21.15.195:0/2537369625 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbccc060440 con 0x7fbccc0f5410 2023-10-12T22:32:59.843 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.839+0000 7fbcbeffd700 1 -- 172.21.15.195:0/2537369625 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcc0013f00 con 0x7fbccc0f5410 2023-10-12T22:32:59.844 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.839+0000 7fbcbeffd700 1 -- 172.21.15.195:0/2537369625 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcc0014d90 con 0x7fbccc0f5410 2023-10-12T22:32:59.844 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.839+0000 7fbcd4496700 1 -- 172.21.15.195:0/2537369625 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc109ae0 con 0x7fbccc0f5410 2023-10-12T22:32:59.845 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.839+0000 7fbcbeffd700 1 -- 172.21.15.195:0/2537369625 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcc00095f0 con 0x7fbccc0f5410 2023-10-12T22:32:59.845 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.839+0000 7fbcd4496700 1 -- 172.21.15.195:0/2537369625 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbccc045ad0 con 0x7fbccc0f5410 2023-10-12T22:32:59.845 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.839+0000 7fbcbeffd700 1 --2- 172.21.15.195:0/2537369625 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbcb802ff60 0x7fbcb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:32:59.845 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.840+0000 7fbcbeffd700 1 -- 172.21.15.195:0/2537369625 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcc0041000 con 0x7fbccc0f5410 2023-10-12T22:32:59.846 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.840+0000 7fbcd1a31700 1 --2- 172.21.15.195:0/2537369625 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbcb802ff60 0x7fbcb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:32:59.846 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.840+0000 7fbcd1a31700 1 --2- 172.21.15.195:0/2537369625 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbcb802ff60 0x7fbcb8032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fbcc80096a0 tx=0x7fbcc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:32:59.846 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:32:59.844+0000 7fbcbeffd700 1 -- 172.21.15.195:0/2537369625 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcc0014420 con 0x7fbccc0f5410 2023-10-12T22:33:00.050 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:00.049+0000 7fbcd4496700 1 -- 172.21.15.195:0/2537369625 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbccc0fae60 con 0x7fbccc0f5410 2023-10-12T22:33:00.051 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:00.050+0000 7fbcbeffd700 1 -- 172.21.15.195:0/2537369625 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbcc001d7f0 con 0x7fbccc0f5410 2023-10-12T22:33:00.052 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:00.052 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:00.053 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:00.052+0000 7fbcd4496700 1 -- 172.21.15.195:0/2537369625 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbcb802ff60 msgr2=0x7fbcb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:00.053 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:00.052+0000 7fbcd4496700 1 --2- 172.21.15.195:0/2537369625 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbcb802ff60 0x7fbcb8032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fbcc80096a0 tx=0x7fbcc8006b40).stop 2023-10-12T22:33:00.053 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:00.052+0000 7fbcd4496700 1 -- 172.21.15.195:0/2537369625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbccc0f5410 msgr2=0x7fbccc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:00.054 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:00.052+0000 7fbcd4496700 1 --2- 172.21.15.195:0/2537369625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbccc0f5410 0x7fbccc109430 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fbcc0013510 tx=0x7fbcc000b040).stop 2023-10-12T22:33:00.054 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:00.052+0000 7fbcd4496700 1 -- 172.21.15.195:0/2537369625 shutdown_connections 2023-10-12T22:33:00.054 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:00.052+0000 7fbcd4496700 1 --2- 172.21.15.195:0/2537369625 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbcb802ff60 0x7fbcb8032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:00.054 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:00.052+0000 7fbcd4496700 1 --2- 172.21.15.195:0/2537369625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbccc0f5410 0x7fbccc109430 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:00.054 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:00.052+0000 7fbcd4496700 1 -- 172.21.15.195:0/2537369625 >> 172.21.15.195:0/2537369625 conn(0x7fbccc0f0b70 msgr2=0x7fbccc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:00.055 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:00.052+0000 7fbcd4496700 1 -- 172.21.15.195:0/2537369625 shutdown_connections 2023-10-12T22:33:00.055 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:00.052+0000 7fbcd4496700 1 -- 172.21.15.195:0/2537369625 wait complete. 2023-10-12T22:33:00.055 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:00.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:32:59 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:32:58.809753+0000 mgr.smithi131.efktma (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:01.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:00.051409+0000 mon.smithi131 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.195:0/2537369625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:01.443 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:01.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:02.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:01 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:00.810116+0000 mgr.smithi131.efktma (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:02.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.570+0000 7f092d9ea700 1 -- 172.21.15.195:0/3609778157 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09280f5410 msgr2=0x7f09280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:02.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.570+0000 7f092d9ea700 1 --2- 172.21.15.195:0/3609778157 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09280f5410 0x7f09280f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0910009fb0 tx=0x7f0910009b30).stop 2023-10-12T22:33:02.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.570+0000 7f092d9ea700 1 -- 172.21.15.195:0/3609778157 shutdown_connections 2023-10-12T22:33:02.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.570+0000 7f092d9ea700 1 --2- 172.21.15.195:0/3609778157 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09280f5410 0x7f09280f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:02.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.570+0000 7f092d9ea700 1 -- 172.21.15.195:0/3609778157 >> 172.21.15.195:0/3609778157 conn(0x7f09280f0b70 msgr2=0x7f09280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:02.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.570+0000 7f092d9ea700 1 -- 172.21.15.195:0/3609778157 shutdown_connections 2023-10-12T22:33:02.573 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.570+0000 7f092d9ea700 1 -- 172.21.15.195:0/3609778157 wait complete. 2023-10-12T22:33:02.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.570+0000 7f092d9ea700 1 Processor -- start 2023-10-12T22:33:02.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.571+0000 7f092d9ea700 1 -- start start 2023-10-12T22:33:02.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.571+0000 7f092d9ea700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09280f5410 0x7f0928109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:02.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.571+0000 7f092d9ea700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0928060240 con 0x7f09280f5410 2023-10-12T22:33:02.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.571+0000 7f0926ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09280f5410 0x7f0928109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:02.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.571+0000 7f0926ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09280f5410 0x7f0928109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:35062/0 (socket says 172.21.15.195:35062) 2023-10-12T22:33:02.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.571+0000 7f0926ffd700 1 -- 172.21.15.195:0/2711812696 learned_addr learned my addr 172.21.15.195:0/2711812696 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:02.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.571+0000 7f0926ffd700 1 -- 172.21.15.195:0/2711812696 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0910005040 con 0x7f09280f5410 2023-10-12T22:33:02.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.571+0000 7f0926ffd700 1 --2- 172.21.15.195:0/2711812696 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09280f5410 0x7f0928109400 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f0910013510 tx=0x7f091000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:02.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.572+0000 7f091f7fe700 1 -- 172.21.15.195:0/2711812696 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0910009320 con 0x7f09280f5410 2023-10-12T22:33:02.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.572+0000 7f092d9ea700 1 -- 172.21.15.195:0/2711812696 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0928060440 con 0x7f09280f5410 2023-10-12T22:33:02.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.572+0000 7f091f7fe700 1 -- 172.21.15.195:0/2711812696 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0910013f00 con 0x7f09280f5410 2023-10-12T22:33:02.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.572+0000 7f091f7fe700 1 -- 172.21.15.195:0/2711812696 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0910014d90 con 0x7f09280f5410 2023-10-12T22:33:02.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.572+0000 7f092d9ea700 1 -- 172.21.15.195:0/2711812696 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0928109ab0 con 0x7f09280f5410 2023-10-12T22:33:02.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.572+0000 7f091f7fe700 1 -- 172.21.15.195:0/2711812696 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09100095f0 con 0x7f09280f5410 2023-10-12T22:33:02.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.573+0000 7f092d9ea700 1 -- 172.21.15.195:0/2711812696 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0928045ad0 con 0x7f09280f5410 2023-10-12T22:33:02.578 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.573+0000 7f091f7fe700 1 --2- 172.21.15.195:0/2711812696 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f091402ffb0 0x7f0914032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:02.578 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.573+0000 7f091f7fe700 1 -- 172.21.15.195:0/2711812696 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0910041050 con 0x7f09280f5410 2023-10-12T22:33:02.578 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.573+0000 7f09267fc700 1 --2- 172.21.15.195:0/2711812696 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f091402ffb0 0x7f0914032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:02.578 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.573+0000 7f09267fc700 1 --2- 172.21.15.195:0/2711812696 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f091402ffb0 0x7f0914032470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f09180096a0 tx=0x7f0918006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:02.579 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.577+0000 7f091f7fe700 1 -- 172.21.15.195:0/2711812696 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0910014420 con 0x7f09280f5410 2023-10-12T22:33:02.789 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.788+0000 7f092d9ea700 1 -- 172.21.15.195:0/2711812696 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09280fae60 con 0x7f09280f5410 2023-10-12T22:33:02.790 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.788+0000 7f091f7fe700 1 -- 172.21.15.195:0/2711812696 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0910010230 con 0x7f09280f5410 2023-10-12T22:33:02.790 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:02.790 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:02.791 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.790+0000 7f092d9ea700 1 -- 172.21.15.195:0/2711812696 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f091402ffb0 msgr2=0x7f0914032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:02.792 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.790+0000 7f092d9ea700 1 --2- 172.21.15.195:0/2711812696 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f091402ffb0 0x7f0914032470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f09180096a0 tx=0x7f0918006b40).stop 2023-10-12T22:33:02.792 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.790+0000 7f092d9ea700 1 -- 172.21.15.195:0/2711812696 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09280f5410 msgr2=0x7f0928109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:02.792 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.790+0000 7f092d9ea700 1 --2- 172.21.15.195:0/2711812696 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09280f5410 0x7f0928109400 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f0910013510 tx=0x7f091000b040).stop 2023-10-12T22:33:02.792 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.790+0000 7f092d9ea700 1 -- 172.21.15.195:0/2711812696 shutdown_connections 2023-10-12T22:33:02.792 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.790+0000 7f092d9ea700 1 --2- 172.21.15.195:0/2711812696 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f091402ffb0 0x7f0914032470 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:02.793 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.790+0000 7f092d9ea700 1 --2- 172.21.15.195:0/2711812696 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09280f5410 0x7f0928109400 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:02.793 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.790+0000 7f092d9ea700 1 -- 172.21.15.195:0/2711812696 >> 172.21.15.195:0/2711812696 conn(0x7f09280f0b70 msgr2=0x7f09280f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:02.793 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.790+0000 7f092d9ea700 1 -- 172.21.15.195:0/2711812696 shutdown_connections 2023-10-12T22:33:02.793 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:02.790+0000 7f092d9ea700 1 -- 172.21.15.195:0/2711812696 wait complete. 2023-10-12T22:33:02.794 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:03.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:02 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:02.789676+0000 mon.smithi131 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.195:0/2711812696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:04.174 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:04.174 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:04.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:03 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:02.810417+0000 mgr.smithi131.efktma (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:05.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.315+0000 7f07fd59a700 1 -- 172.21.15.195:0/1625526885 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07f00138a0 con 0x7f08000f5410 2023-10-12T22:33:05.318 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.316+0000 7f080497b700 1 -- 172.21.15.195:0/1625526885 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08000f5410 msgr2=0x7f08000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:05.318 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.316+0000 7f080497b700 1 --2- 172.21.15.195:0/1625526885 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08000f5410 0x7f08000f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f07f0009fb0 tx=0x7f07f0009b30).stop 2023-10-12T22:33:05.318 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.316+0000 7f080497b700 1 -- 172.21.15.195:0/1625526885 shutdown_connections 2023-10-12T22:33:05.318 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.316+0000 7f080497b700 1 --2- 172.21.15.195:0/1625526885 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08000f5410 0x7f08000f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:05.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.316+0000 7f080497b700 1 -- 172.21.15.195:0/1625526885 >> 172.21.15.195:0/1625526885 conn(0x7f08000f0b70 msgr2=0x7f08000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:05.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.316+0000 7f080497b700 1 -- 172.21.15.195:0/1625526885 shutdown_connections 2023-10-12T22:33:05.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.316+0000 7f080497b700 1 -- 172.21.15.195:0/1625526885 wait complete. 2023-10-12T22:33:05.321 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.317+0000 7f080497b700 1 Processor -- start 2023-10-12T22:33:05.321 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.317+0000 7f080497b700 1 -- start start 2023-10-12T22:33:05.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.317+0000 7f080497b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08000f5410 0x7f08000fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:05.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.317+0000 7f080497b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08000fb820 con 0x7f08000f5410 2023-10-12T22:33:05.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.317+0000 7f07fe59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08000f5410 0x7f08000fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:05.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.317+0000 7f07fe59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08000f5410 0x7f08000fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:35084/0 (socket says 172.21.15.195:35084) 2023-10-12T22:33:05.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.317+0000 7f07fe59c700 1 -- 172.21.15.195:0/308124807 learned_addr learned my addr 172.21.15.195:0/308124807 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:05.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.317+0000 7f07fe59c700 1 -- 172.21.15.195:0/308124807 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f0005040 con 0x7f08000f5410 2023-10-12T22:33:05.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.318+0000 7f07fe59c700 1 --2- 172.21.15.195:0/308124807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08000f5410 0x7f08000fd3a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f07f0013510 tx=0x7f07f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:05.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.318+0000 7f07f6ffd700 1 -- 172.21.15.195:0/308124807 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07f0009320 con 0x7f08000f5410 2023-10-12T22:33:05.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.318+0000 7f080497b700 1 -- 172.21.15.195:0/308124807 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08000fb9c0 con 0x7f08000f5410 2023-10-12T22:33:05.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.318+0000 7f07f6ffd700 1 -- 172.21.15.195:0/308124807 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07f0013f00 con 0x7f08000f5410 2023-10-12T22:33:05.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.318+0000 7f07f6ffd700 1 -- 172.21.15.195:0/308124807 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07f0014d90 con 0x7f08000f5410 2023-10-12T22:33:05.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.318+0000 7f080497b700 1 -- 172.21.15.195:0/308124807 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08000fbe50 con 0x7f08000f5410 2023-10-12T22:33:05.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.319+0000 7f07f6ffd700 1 -- 172.21.15.195:0/308124807 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07f00095f0 con 0x7f08000f5410 2023-10-12T22:33:05.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.319+0000 7f080497b700 1 -- 172.21.15.195:0/308124807 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0800059100 con 0x7f08000f5410 2023-10-12T22:33:05.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.319+0000 7f07f6ffd700 1 --2- 172.21.15.195:0/308124807 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f07ec02ff60 0x7f07ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:05.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.319+0000 7f07f6ffd700 1 -- 172.21.15.195:0/308124807 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07f0040fe0 con 0x7f08000f5410 2023-10-12T22:33:05.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.319+0000 7f07fdd9b700 1 --2- 172.21.15.195:0/308124807 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f07ec02ff60 0x7f07ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:05.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.320+0000 7f07fdd9b700 1 --2- 172.21.15.195:0/308124807 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f07ec02ff60 0x7f07ec032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f07e80096a0 tx=0x7f07e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:05.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.324+0000 7f07f6ffd700 1 -- 172.21.15.195:0/308124807 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07f001d7f0 con 0x7f08000f5410 2023-10-12T22:33:05.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.530+0000 7f080497b700 1 -- 172.21.15.195:0/308124807 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08000fae60 con 0x7f08000f5410 2023-10-12T22:33:05.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.531+0000 7f07f6ffd700 1 -- 172.21.15.195:0/308124807 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f07f0010230 con 0x7f08000f5410 2023-10-12T22:33:05.533 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:05.533 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:05.534 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.533+0000 7f080497b700 1 -- 172.21.15.195:0/308124807 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f07ec02ff60 msgr2=0x7f07ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:05.534 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.533+0000 7f080497b700 1 --2- 172.21.15.195:0/308124807 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f07ec02ff60 0x7f07ec032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f07e80096a0 tx=0x7f07e8006b40).stop 2023-10-12T22:33:05.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.533+0000 7f080497b700 1 -- 172.21.15.195:0/308124807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08000f5410 msgr2=0x7f08000fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:05.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.533+0000 7f080497b700 1 --2- 172.21.15.195:0/308124807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08000f5410 0x7f08000fd3a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f07f0013510 tx=0x7f07f000b040).stop 2023-10-12T22:33:05.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.533+0000 7f080497b700 1 -- 172.21.15.195:0/308124807 shutdown_connections 2023-10-12T22:33:05.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.533+0000 7f080497b700 1 --2- 172.21.15.195:0/308124807 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f07ec02ff60 0x7f07ec032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:05.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.533+0000 7f080497b700 1 --2- 172.21.15.195:0/308124807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08000f5410 0x7f08000fd3a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:05.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.533+0000 7f080497b700 1 -- 172.21.15.195:0/308124807 >> 172.21.15.195:0/308124807 conn(0x7f08000f0b70 msgr2=0x7f080005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:05.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.533+0000 7f080497b700 1 -- 172.21.15.195:0/308124807 shutdown_connections 2023-10-12T22:33:05.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:05.533+0000 7f080497b700 1 -- 172.21.15.195:0/308124807 wait complete. 2023-10-12T22:33:05.537 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:06.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:05 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:04.810715+0000 mgr.smithi131.efktma (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:06.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:05 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:05.532613+0000 mon.smithi131 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.195:0/308124807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:06.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:06.952 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:08.114 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.112+0000 7f4246dbd700 1 -- 172.21.15.195:0/325904804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42400f5410 msgr2=0x7f42400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:08.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.112+0000 7f4246dbd700 1 --2- 172.21.15.195:0/325904804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42400f5410 0x7f42400f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f4234009fb0 tx=0x7f4234009b30).stop 2023-10-12T22:33:08.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.112+0000 7f4246dbd700 1 -- 172.21.15.195:0/325904804 shutdown_connections 2023-10-12T22:33:08.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.112+0000 7f4246dbd700 1 --2- 172.21.15.195:0/325904804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42400f5410 0x7f42400f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:08.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.112+0000 7f4246dbd700 1 -- 172.21.15.195:0/325904804 >> 172.21.15.195:0/325904804 conn(0x7f42400f0b70 msgr2=0x7f42400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:08.116 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.112+0000 7f4246dbd700 1 -- 172.21.15.195:0/325904804 shutdown_connections 2023-10-12T22:33:08.116 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.112+0000 7f4246dbd700 1 -- 172.21.15.195:0/325904804 wait complete. 2023-10-12T22:33:08.116 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.113+0000 7f4246dbd700 1 Processor -- start 2023-10-12T22:33:08.116 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.113+0000 7f4246dbd700 1 -- start start 2023-10-12T22:33:08.116 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.113+0000 7f4246dbd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42400f5410 0x7f4240109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:08.117 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.113+0000 7f4246dbd700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4240060240 con 0x7f42400f5410 2023-10-12T22:33:08.117 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.113+0000 7f4244b59700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42400f5410 0x7f4240109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:08.117 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.113+0000 7f4244b59700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42400f5410 0x7f4240109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:35096/0 (socket says 172.21.15.195:35096) 2023-10-12T22:33:08.117 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.113+0000 7f4244b59700 1 -- 172.21.15.195:0/1910897648 learned_addr learned my addr 172.21.15.195:0/1910897648 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:08.119 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.114+0000 7f4244b59700 1 -- 172.21.15.195:0/1910897648 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4234005040 con 0x7f42400f5410 2023-10-12T22:33:08.119 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.114+0000 7f4244b59700 1 --2- 172.21.15.195:0/1910897648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42400f5410 0x7f4240109430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f4234014000 tx=0x7f423400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:08.119 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.114+0000 7f423d7fa700 1 -- 172.21.15.195:0/1910897648 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42340095f0 con 0x7f42400f5410 2023-10-12T22:33:08.119 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.114+0000 7f4246dbd700 1 -- 172.21.15.195:0/1910897648 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4240060440 con 0x7f42400f5410 2023-10-12T22:33:08.119 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.114+0000 7f423d7fa700 1 -- 172.21.15.195:0/1910897648 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4234007e70 con 0x7f42400f5410 2023-10-12T22:33:08.120 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.114+0000 7f423d7fa700 1 -- 172.21.15.195:0/1910897648 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4234014ea0 con 0x7f42400f5410 2023-10-12T22:33:08.120 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.114+0000 7f4246dbd700 1 -- 172.21.15.195:0/1910897648 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4240109ae0 con 0x7f42400f5410 2023-10-12T22:33:08.120 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.115+0000 7f423d7fa700 1 -- 172.21.15.195:0/1910897648 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4234025070 con 0x7f42400f5410 2023-10-12T22:33:08.120 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.115+0000 7f423d7fa700 1 --2- 172.21.15.195:0/1910897648 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f423002ff60 0x7f4230032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:08.120 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.115+0000 7f4246dbd700 1 -- 172.21.15.195:0/1910897648 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4240045ad0 con 0x7f42400f5410 2023-10-12T22:33:08.120 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.115+0000 7f423d7fa700 1 -- 172.21.15.195:0/1910897648 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4234040980 con 0x7f42400f5410 2023-10-12T22:33:08.121 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.115+0000 7f423ffff700 1 --2- 172.21.15.195:0/1910897648 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f423002ff60 0x7f4230032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:08.121 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.116+0000 7f423ffff700 1 --2- 172.21.15.195:0/1910897648 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f423002ff60 0x7f4230032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f422c0096a0 tx=0x7f422c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:08.121 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.120+0000 7f423d7fa700 1 -- 172.21.15.195:0/1910897648 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4234026350 con 0x7f42400f5410 2023-10-12T22:33:08.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:06.811072+0000 mgr.smithi131.efktma (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:08.326 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.324+0000 7f4246dbd700 1 -- 172.21.15.195:0/1910897648 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42400faf50 con 0x7f42400f5410 2023-10-12T22:33:08.327 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.325+0000 7f423d7fa700 1 -- 172.21.15.195:0/1910897648 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4234012330 con 0x7f42400f5410 2023-10-12T22:33:08.327 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:08.328 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:08.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.327+0000 7f4246dbd700 1 -- 172.21.15.195:0/1910897648 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f423002ff60 msgr2=0x7f4230032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:08.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.327+0000 7f4246dbd700 1 --2- 172.21.15.195:0/1910897648 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f423002ff60 0x7f4230032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f422c0096a0 tx=0x7f422c006b40).stop 2023-10-12T22:33:08.330 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.327+0000 7f4246dbd700 1 -- 172.21.15.195:0/1910897648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42400f5410 msgr2=0x7f4240109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:08.330 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.327+0000 7f4246dbd700 1 --2- 172.21.15.195:0/1910897648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42400f5410 0x7f4240109430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f4234014000 tx=0x7f423400b040).stop 2023-10-12T22:33:08.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.327+0000 7f4246dbd700 1 -- 172.21.15.195:0/1910897648 shutdown_connections 2023-10-12T22:33:08.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.327+0000 7f4246dbd700 1 --2- 172.21.15.195:0/1910897648 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f423002ff60 0x7f4230032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:08.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.327+0000 7f4246dbd700 1 --2- 172.21.15.195:0/1910897648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42400f5410 0x7f4240109430 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:08.332 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.327+0000 7f4246dbd700 1 -- 172.21.15.195:0/1910897648 >> 172.21.15.195:0/1910897648 conn(0x7f42400f0b70 msgr2=0x7f42400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:08.332 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.327+0000 7f4246dbd700 1 -- 172.21.15.195:0/1910897648 shutdown_connections 2023-10-12T22:33:08.332 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:08.327+0000 7f4246dbd700 1 -- 172.21.15.195:0/1910897648 wait complete. 2023-10-12T22:33:08.333 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:09.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:08.326509+0000 mon.smithi131 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.195:0/1910897648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:09.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:09.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:10.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:09 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:08.811457+0000 mgr.smithi131.efktma (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:10.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.869+0000 7f0993fff700 1 -- 172.21.15.195:0/1228398623 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09940f4920 msgr2=0x7f09940f4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:10.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.869+0000 7f0993fff700 1 --2- 172.21.15.195:0/1228398623 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09940f4920 0x7f09940f4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f0984009fb0 tx=0x7f0984009b30).stop 2023-10-12T22:33:10.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.869+0000 7f0993fff700 1 -- 172.21.15.195:0/1228398623 shutdown_connections 2023-10-12T22:33:10.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.869+0000 7f0993fff700 1 --2- 172.21.15.195:0/1228398623 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09940f4920 0x7f09940f4d40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:10.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.869+0000 7f0993fff700 1 -- 172.21.15.195:0/1228398623 >> 172.21.15.195:0/1228398623 conn(0x7f09940f0120 msgr2=0x7f09940f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:10.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.869+0000 7f0993fff700 1 -- 172.21.15.195:0/1228398623 shutdown_connections 2023-10-12T22:33:10.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.869+0000 7f0993fff700 1 -- 172.21.15.195:0/1228398623 wait complete. 2023-10-12T22:33:10.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.869+0000 7f0993fff700 1 Processor -- start 2023-10-12T22:33:10.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.870+0000 7f0993fff700 1 -- start start 2023-10-12T22:33:10.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.870+0000 7f0993fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09940f4920 0x7f09941064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:10.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.870+0000 7f0993fff700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09941069e0 con 0x7f09940f4920 2023-10-12T22:33:10.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.870+0000 7f0992ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09940f4920 0x7f09941064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:10.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.870+0000 7f0992ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09940f4920 0x7f09941064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:39372/0 (socket says 172.21.15.195:39372) 2023-10-12T22:33:10.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.870+0000 7f0992ffd700 1 -- 172.21.15.195:0/57150765 learned_addr learned my addr 172.21.15.195:0/57150765 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:10.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.870+0000 7f0992ffd700 1 -- 172.21.15.195:0/57150765 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0984005040 con 0x7f09940f4920 2023-10-12T22:33:10.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.870+0000 7f0992ffd700 1 --2- 172.21.15.195:0/57150765 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09940f4920 0x7f09941064f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f0984013510 tx=0x7f098400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:10.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.870+0000 7f097b7fe700 1 -- 172.21.15.195:0/57150765 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0984009320 con 0x7f09940f4920 2023-10-12T22:33:10.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.870+0000 7f0993fff700 1 -- 172.21.15.195:0/57150765 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0994106be0 con 0x7f09940f4920 2023-10-12T22:33:10.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.871+0000 7f0993fff700 1 -- 172.21.15.195:0/57150765 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0994107070 con 0x7f09940f4920 2023-10-12T22:33:10.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.871+0000 7f097b7fe700 1 -- 172.21.15.195:0/57150765 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0984013f00 con 0x7f09940f4920 2023-10-12T22:33:10.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.871+0000 7f097b7fe700 1 -- 172.21.15.195:0/57150765 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0984014dc0 con 0x7f09940f4920 2023-10-12T22:33:10.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.871+0000 7f097b7fe700 1 -- 172.21.15.195:0/57150765 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f098401d3d0 con 0x7f09940f4920 2023-10-12T22:33:10.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.871+0000 7f097b7fe700 1 --2- 172.21.15.195:0/57150765 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f097c02ff60 0x7f097c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:10.878 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.871+0000 7f0993fff700 1 -- 172.21.15.195:0/57150765 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0994045ad0 con 0x7f09940f4920 2023-10-12T22:33:10.878 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.872+0000 7f097b7fe700 1 -- 172.21.15.195:0/57150765 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0984041120 con 0x7f09940f4920 2023-10-12T22:33:10.878 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.872+0000 7f09927fc700 1 --2- 172.21.15.195:0/57150765 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f097c02ff60 0x7f097c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:10.878 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.872+0000 7f09927fc700 1 --2- 172.21.15.195:0/57150765 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f097c02ff60 0x7f097c032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f09880096a0 tx=0x7f0988006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:10.879 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:10.876+0000 7f097b7fe700 1 -- 172.21.15.195:0/57150765 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09840263a0 con 0x7f09940f4920 2023-10-12T22:33:11.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:11.083+0000 7f0993fff700 1 -- 172.21.15.195:0/57150765 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09941004d0 con 0x7f09940f4920 2023-10-12T22:33:11.085 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:11.084+0000 7f097b7fe700 1 -- 172.21.15.195:0/57150765 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f09840126e0 con 0x7f09940f4920 2023-10-12T22:33:11.085 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:11.085 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:11.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:11.085+0000 7f0993fff700 1 -- 172.21.15.195:0/57150765 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f097c02ff60 msgr2=0x7f097c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:11.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:11.085+0000 7f0993fff700 1 --2- 172.21.15.195:0/57150765 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f097c02ff60 0x7f097c032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f09880096a0 tx=0x7f0988006b40).stop 2023-10-12T22:33:11.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:11.085+0000 7f0993fff700 1 -- 172.21.15.195:0/57150765 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09940f4920 msgr2=0x7f09941064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:11.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:11.085+0000 7f0993fff700 1 --2- 172.21.15.195:0/57150765 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09940f4920 0x7f09941064f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f0984013510 tx=0x7f098400b040).stop 2023-10-12T22:33:11.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:11.085+0000 7f0993fff700 1 -- 172.21.15.195:0/57150765 shutdown_connections 2023-10-12T22:33:11.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:11.085+0000 7f0993fff700 1 --2- 172.21.15.195:0/57150765 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f097c02ff60 0x7f097c032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:11.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:11.085+0000 7f0993fff700 1 --2- 172.21.15.195:0/57150765 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09940f4920 0x7f09941064f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:11.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:11.085+0000 7f0993fff700 1 -- 172.21.15.195:0/57150765 >> 172.21.15.195:0/57150765 conn(0x7f09940f0120 msgr2=0x7f09940f2380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:11.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:11.085+0000 7f0993fff700 1 -- 172.21.15.195:0/57150765 shutdown_connections 2023-10-12T22:33:11.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:11.085+0000 7f0993fff700 1 -- 172.21.15.195:0/57150765 wait complete. 2023-10-12T22:33:11.089 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:12.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:10.811824+0000 mgr.smithi131.efktma (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:12.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:11.085204+0000 mon.smithi131 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.195:0/57150765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:12.496 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:12.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:13.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.612+0000 7fdf3b271700 1 -- 172.21.15.195:0/1622786745 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf340f5410 msgr2=0x7fdf340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:13.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.612+0000 7fdf3b271700 1 --2- 172.21.15.195:0/1622786745 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf340f5410 0x7fdf340f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fdf28009fb0 tx=0x7fdf28009b30).stop 2023-10-12T22:33:13.615 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.612+0000 7fdf3b271700 1 -- 172.21.15.195:0/1622786745 shutdown_connections 2023-10-12T22:33:13.615 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.612+0000 7fdf3b271700 1 --2- 172.21.15.195:0/1622786745 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf340f5410 0x7fdf340f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:13.615 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.612+0000 7fdf3b271700 1 -- 172.21.15.195:0/1622786745 >> 172.21.15.195:0/1622786745 conn(0x7fdf340f0b70 msgr2=0x7fdf340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:13.615 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.612+0000 7fdf3b271700 1 -- 172.21.15.195:0/1622786745 shutdown_connections 2023-10-12T22:33:13.615 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.612+0000 7fdf3b271700 1 -- 172.21.15.195:0/1622786745 wait complete. 2023-10-12T22:33:13.616 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.613+0000 7fdf3b271700 1 Processor -- start 2023-10-12T22:33:13.616 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.613+0000 7fdf3b271700 1 -- start start 2023-10-12T22:33:13.616 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.613+0000 7fdf3b271700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf340f5410 0x7fdf34105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:13.616 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.613+0000 7fdf3b271700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf34105b20 con 0x7fdf340f5410 2023-10-12T22:33:13.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.613+0000 7fdf3900d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf340f5410 0x7fdf34105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:13.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.613+0000 7fdf3900d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf340f5410 0x7fdf34105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:39392/0 (socket says 172.21.15.195:39392) 2023-10-12T22:33:13.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.613+0000 7fdf3900d700 1 -- 172.21.15.195:0/1666280442 learned_addr learned my addr 172.21.15.195:0/1666280442 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:13.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.614+0000 7fdf3900d700 1 -- 172.21.15.195:0/1666280442 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf28005040 con 0x7fdf340f5410 2023-10-12T22:33:13.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.614+0000 7fdf3900d700 1 --2- 172.21.15.195:0/1666280442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf340f5410 0x7fdf34105630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fdf28013510 tx=0x7fdf2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:13.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.614+0000 7fdf25ffb700 1 -- 172.21.15.195:0/1666280442 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf28009320 con 0x7fdf340f5410 2023-10-12T22:33:13.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.614+0000 7fdf25ffb700 1 -- 172.21.15.195:0/1666280442 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf28013f00 con 0x7fdf340f5410 2023-10-12T22:33:13.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.614+0000 7fdf25ffb700 1 -- 172.21.15.195:0/1666280442 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf28014d90 con 0x7fdf340f5410 2023-10-12T22:33:13.620 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.614+0000 7fdf3b271700 1 -- 172.21.15.195:0/1666280442 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf34105d20 con 0x7fdf340f5410 2023-10-12T22:33:13.620 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.614+0000 7fdf3b271700 1 -- 172.21.15.195:0/1666280442 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf341061b0 con 0x7fdf340f5410 2023-10-12T22:33:13.620 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.615+0000 7fdf25ffb700 1 -- 172.21.15.195:0/1666280442 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf280095f0 con 0x7fdf340f5410 2023-10-12T22:33:13.620 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.615+0000 7fdf3b271700 1 -- 172.21.15.195:0/1666280442 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf34045ad0 con 0x7fdf340f5410 2023-10-12T22:33:13.620 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.615+0000 7fdf25ffb700 1 --2- 172.21.15.195:0/1666280442 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf2002ffb0 0x7fdf20032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:13.621 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.615+0000 7fdf25ffb700 1 -- 172.21.15.195:0/1666280442 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf28040f90 con 0x7fdf340f5410 2023-10-12T22:33:13.621 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.615+0000 7fdf3880c700 1 --2- 172.21.15.195:0/1666280442 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf2002ffb0 0x7fdf20032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:13.621 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.616+0000 7fdf3880c700 1 --2- 172.21.15.195:0/1666280442 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf2002ffb0 0x7fdf20032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fdf300096a0 tx=0x7fdf30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:13.622 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.620+0000 7fdf25ffb700 1 -- 172.21.15.195:0/1666280442 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf28014420 con 0x7fdf340f5410 2023-10-12T22:33:13.829 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.828+0000 7fdf3b271700 1 -- 172.21.15.195:0/1666280442 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf340fae60 con 0x7fdf340f5410 2023-10-12T22:33:13.830 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.829+0000 7fdf25ffb700 1 -- 172.21.15.195:0/1666280442 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdf2801d7f0 con 0x7fdf340f5410 2023-10-12T22:33:13.830 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:13.830 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:13.833 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.830+0000 7fdf3b271700 1 -- 172.21.15.195:0/1666280442 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf2002ffb0 msgr2=0x7fdf20032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:13.835 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.830+0000 7fdf3b271700 1 --2- 172.21.15.195:0/1666280442 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf2002ffb0 0x7fdf20032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fdf300096a0 tx=0x7fdf30006b40).stop 2023-10-12T22:33:13.835 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.830+0000 7fdf3b271700 1 -- 172.21.15.195:0/1666280442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf340f5410 msgr2=0x7fdf34105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:13.835 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.830+0000 7fdf3b271700 1 --2- 172.21.15.195:0/1666280442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf340f5410 0x7fdf34105630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fdf28013510 tx=0x7fdf2800b040).stop 2023-10-12T22:33:13.835 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.830+0000 7fdf3b271700 1 -- 172.21.15.195:0/1666280442 shutdown_connections 2023-10-12T22:33:13.836 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.830+0000 7fdf3b271700 1 --2- 172.21.15.195:0/1666280442 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf2002ffb0 0x7fdf20032470 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:13.836 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.830+0000 7fdf3b271700 1 --2- 172.21.15.195:0/1666280442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf340f5410 0x7fdf34105630 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:13.836 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.830+0000 7fdf3b271700 1 -- 172.21.15.195:0/1666280442 >> 172.21.15.195:0/1666280442 conn(0x7fdf340f0b70 msgr2=0x7fdf340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:13.836 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.830+0000 7fdf3b271700 1 -- 172.21.15.195:0/1666280442 shutdown_connections 2023-10-12T22:33:13.836 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:13.830+0000 7fdf3b271700 1 -- 172.21.15.195:0/1666280442 wait complete. 2023-10-12T22:33:13.836 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:14.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:13 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:12.812210+0000 mgr.smithi131.efktma (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:14.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:13 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:13.829851+0000 mon.smithi131 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.195:0/1666280442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:15.241 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:15.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:16.374 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.372+0000 7f7b5715d700 1 -- 172.21.15.195:0/2231336428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7b500f5410 msgr2=0x7f7b500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:16.375 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.372+0000 7f7b5715d700 1 --2- 172.21.15.195:0/2231336428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7b500f5410 0x7f7b500f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f7b44009fb0 tx=0x7f7b44009b30).stop 2023-10-12T22:33:16.375 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.372+0000 7f7b5715d700 1 -- 172.21.15.195:0/2231336428 shutdown_connections 2023-10-12T22:33:16.375 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.372+0000 7f7b5715d700 1 --2- 172.21.15.195:0/2231336428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7b500f5410 0x7f7b500f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:16.375 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.372+0000 7f7b5715d700 1 -- 172.21.15.195:0/2231336428 >> 172.21.15.195:0/2231336428 conn(0x7f7b500f0b70 msgr2=0x7f7b500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:16.376 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.372+0000 7f7b5715d700 1 -- 172.21.15.195:0/2231336428 shutdown_connections 2023-10-12T22:33:16.376 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.372+0000 7f7b5715d700 1 -- 172.21.15.195:0/2231336428 wait complete. 2023-10-12T22:33:16.376 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.373+0000 7f7b5715d700 1 Processor -- start 2023-10-12T22:33:16.376 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.373+0000 7f7b5715d700 1 -- start start 2023-10-12T22:33:16.377 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.373+0000 7f7b5715d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7b500f5410 0x7f7b50105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:16.377 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.373+0000 7f7b5715d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b50105b20 con 0x7f7b500f5410 2023-10-12T22:33:16.377 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.373+0000 7f7b54ef9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7b500f5410 0x7f7b50105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:16.377 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.373+0000 7f7b54ef9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7b500f5410 0x7f7b50105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:39404/0 (socket says 172.21.15.195:39404) 2023-10-12T22:33:16.377 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.373+0000 7f7b54ef9700 1 -- 172.21.15.195:0/1899225779 learned_addr learned my addr 172.21.15.195:0/1899225779 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:16.378 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.374+0000 7f7b54ef9700 1 -- 172.21.15.195:0/1899225779 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44005040 con 0x7f7b500f5410 2023-10-12T22:33:16.379 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.374+0000 7f7b54ef9700 1 --2- 172.21.15.195:0/1899225779 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7b500f5410 0x7f7b50105630 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f7b44013510 tx=0x7f7b4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:16.379 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.374+0000 7f7b4d7fa700 1 -- 172.21.15.195:0/1899225779 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b44009320 con 0x7f7b500f5410 2023-10-12T22:33:16.380 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.374+0000 7f7b5715d700 1 -- 172.21.15.195:0/1899225779 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b50105d20 con 0x7f7b500f5410 2023-10-12T22:33:16.380 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.374+0000 7f7b4d7fa700 1 -- 172.21.15.195:0/1899225779 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b44013f00 con 0x7f7b500f5410 2023-10-12T22:33:16.380 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.374+0000 7f7b4d7fa700 1 -- 172.21.15.195:0/1899225779 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b44014d90 con 0x7f7b500f5410 2023-10-12T22:33:16.380 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.374+0000 7f7b5715d700 1 -- 172.21.15.195:0/1899225779 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b501061b0 con 0x7f7b500f5410 2023-10-12T22:33:16.380 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.375+0000 7f7b4d7fa700 1 -- 172.21.15.195:0/1899225779 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b44014420 con 0x7f7b500f5410 2023-10-12T22:33:16.381 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.375+0000 7f7b4d7fa700 1 --2- 172.21.15.195:0/1899225779 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7b4002ff60 0x7f7b40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:16.381 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.375+0000 7f7b5715d700 1 -- 172.21.15.195:0/1899225779 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b50045ad0 con 0x7f7b500f5410 2023-10-12T22:33:16.381 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.375+0000 7f7b4d7fa700 1 -- 172.21.15.195:0/1899225779 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b440410c0 con 0x7f7b500f5410 2023-10-12T22:33:16.381 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.375+0000 7f7b4ffff700 1 --2- 172.21.15.195:0/1899225779 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7b4002ff60 0x7f7b40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:16.382 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.375+0000 7f7b4ffff700 1 --2- 172.21.15.195:0/1899225779 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7b4002ff60 0x7f7b40032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f7b3c0096a0 tx=0x7f7b3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:16.382 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.379+0000 7f7b4d7fa700 1 -- 172.21.15.195:0/1899225779 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b4406d2b0 con 0x7f7b500f5410 2023-10-12T22:33:16.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:15 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:14.812572+0000 mgr.smithi131.efktma (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:16.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.590+0000 7f7b5715d700 1 -- 172.21.15.195:0/1899225779 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b500faf50 con 0x7f7b500f5410 2023-10-12T22:33:16.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.590+0000 7f7b4d7fa700 1 -- 172.21.15.195:0/1899225779 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7b4401dd50 con 0x7f7b500f5410 2023-10-12T22:33:16.592 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:16.592 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:16.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.592+0000 7f7b5715d700 1 -- 172.21.15.195:0/1899225779 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7b4002ff60 msgr2=0x7f7b40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:16.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.592+0000 7f7b5715d700 1 --2- 172.21.15.195:0/1899225779 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7b4002ff60 0x7f7b40032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f7b3c0096a0 tx=0x7f7b3c006b40).stop 2023-10-12T22:33:16.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.592+0000 7f7b5715d700 1 -- 172.21.15.195:0/1899225779 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7b500f5410 msgr2=0x7f7b50105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:16.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.592+0000 7f7b5715d700 1 --2- 172.21.15.195:0/1899225779 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7b500f5410 0x7f7b50105630 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f7b44013510 tx=0x7f7b4400b040).stop 2023-10-12T22:33:16.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.592+0000 7f7b5715d700 1 -- 172.21.15.195:0/1899225779 shutdown_connections 2023-10-12T22:33:16.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.592+0000 7f7b5715d700 1 --2- 172.21.15.195:0/1899225779 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7b4002ff60 0x7f7b40032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:16.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.592+0000 7f7b5715d700 1 --2- 172.21.15.195:0/1899225779 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7b500f5410 0x7f7b50105630 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:16.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.592+0000 7f7b5715d700 1 -- 172.21.15.195:0/1899225779 >> 172.21.15.195:0/1899225779 conn(0x7f7b500f0b70 msgr2=0x7f7b500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:16.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.592+0000 7f7b5715d700 1 -- 172.21.15.195:0/1899225779 shutdown_connections 2023-10-12T22:33:16.595 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:16.592+0000 7f7b5715d700 1 -- 172.21.15.195:0/1899225779 wait complete. 2023-10-12T22:33:16.595 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:17.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:17 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:16.591661+0000 mon.smithi131 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.195:0/1899225779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:17.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:17.983 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:18.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:18 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:16.812934+0000 mgr.smithi131.efktma (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:19.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.154+0000 7f37a97a8700 1 -- 172.21.15.195:0/2978028272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37a40f51a0 msgr2=0x7f37a40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:19.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.154+0000 7f37a97a8700 1 --2- 172.21.15.195:0/2978028272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37a40f51a0 0x7f37a40f55c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f3794009fb0 tx=0x7f3794009b30).stop 2023-10-12T22:33:19.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.155+0000 7f37a97a8700 1 -- 172.21.15.195:0/2978028272 shutdown_connections 2023-10-12T22:33:19.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.155+0000 7f37a97a8700 1 --2- 172.21.15.195:0/2978028272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37a40f51a0 0x7f37a40f55c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:19.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.155+0000 7f37a97a8700 1 -- 172.21.15.195:0/2978028272 >> 172.21.15.195:0/2978028272 conn(0x7f37a40f09a0 msgr2=0x7f37a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:19.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.155+0000 7f37a97a8700 1 -- 172.21.15.195:0/2978028272 shutdown_connections 2023-10-12T22:33:19.158 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.155+0000 7f37a97a8700 1 -- 172.21.15.195:0/2978028272 wait complete. 2023-10-12T22:33:19.158 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.155+0000 7f37a97a8700 1 Processor -- start 2023-10-12T22:33:19.160 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.155+0000 7f37a97a8700 1 -- start start 2023-10-12T22:33:19.160 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.156+0000 7f37a97a8700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37a40f51a0 0x7f37a41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:19.160 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.156+0000 7f37a97a8700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a41058a0 con 0x7f37a40f51a0 2023-10-12T22:33:19.160 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.156+0000 7f37a3fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37a40f51a0 0x7f37a41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:19.161 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.156+0000 7f37a3fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37a40f51a0 0x7f37a41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:34424/0 (socket says 172.21.15.195:34424) 2023-10-12T22:33:19.161 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.156+0000 7f37a3fff700 1 -- 172.21.15.195:0/2989304849 learned_addr learned my addr 172.21.15.195:0/2989304849 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:19.161 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.156+0000 7f37a3fff700 1 -- 172.21.15.195:0/2989304849 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3794005040 con 0x7f37a40f51a0 2023-10-12T22:33:19.161 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.156+0000 7f37a3fff700 1 --2- 172.21.15.195:0/2989304849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37a40f51a0 0x7f37a41053b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f3794013510 tx=0x7f379400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:19.162 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.157+0000 7f37a0ff9700 1 -- 172.21.15.195:0/2989304849 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3794009320 con 0x7f37a40f51a0 2023-10-12T22:33:19.162 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.157+0000 7f37a97a8700 1 -- 172.21.15.195:0/2989304849 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a4105aa0 con 0x7f37a40f51a0 2023-10-12T22:33:19.162 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.157+0000 7f37a0ff9700 1 -- 172.21.15.195:0/2989304849 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3794013f00 con 0x7f37a40f51a0 2023-10-12T22:33:19.162 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.157+0000 7f37a0ff9700 1 -- 172.21.15.195:0/2989304849 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3794014dc0 con 0x7f37a40f51a0 2023-10-12T22:33:19.162 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.157+0000 7f37a97a8700 1 -- 172.21.15.195:0/2989304849 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a4105f30 con 0x7f37a40f51a0 2023-10-12T22:33:19.163 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.157+0000 7f37a0ff9700 1 -- 172.21.15.195:0/2989304849 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37940095f0 con 0x7f37a40f51a0 2023-10-12T22:33:19.163 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.157+0000 7f37a97a8700 1 -- 172.21.15.195:0/2989304849 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a4045ad0 con 0x7f37a40f51a0 2023-10-12T22:33:19.163 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.158+0000 7f37a0ff9700 1 --2- 172.21.15.195:0/2989304849 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f378c02ff60 0x7f378c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:19.163 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.158+0000 7f37a0ff9700 1 -- 172.21.15.195:0/2989304849 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37940410f0 con 0x7f37a40f51a0 2023-10-12T22:33:19.163 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.158+0000 7f37a37fe700 1 --2- 172.21.15.195:0/2989304849 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f378c02ff60 0x7f378c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:19.164 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.158+0000 7f37a37fe700 1 --2- 172.21.15.195:0/2989304849 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f378c02ff60 0x7f378c032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f37980096a0 tx=0x7f3798006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:19.164 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.162+0000 7f37a0ff9700 1 -- 172.21.15.195:0/2989304849 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3794014420 con 0x7f37a40f51a0 2023-10-12T22:33:19.368 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.367+0000 7f37a97a8700 1 -- 172.21.15.195:0/2989304849 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37a4106de0 con 0x7f37a40f51a0 2023-10-12T22:33:19.369 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.368+0000 7f37a0ff9700 1 -- 172.21.15.195:0/2989304849 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3794010230 con 0x7f37a40f51a0 2023-10-12T22:33:19.369 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:19.370 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:19.370 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.369+0000 7f37a97a8700 1 -- 172.21.15.195:0/2989304849 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f378c02ff60 msgr2=0x7f378c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:19.371 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.369+0000 7f37a97a8700 1 --2- 172.21.15.195:0/2989304849 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f378c02ff60 0x7f378c032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f37980096a0 tx=0x7f3798006b40).stop 2023-10-12T22:33:19.371 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.369+0000 7f37a97a8700 1 -- 172.21.15.195:0/2989304849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37a40f51a0 msgr2=0x7f37a41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:19.371 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.369+0000 7f37a97a8700 1 --2- 172.21.15.195:0/2989304849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37a40f51a0 0x7f37a41053b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f3794013510 tx=0x7f379400b040).stop 2023-10-12T22:33:19.371 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.369+0000 7f37a97a8700 1 -- 172.21.15.195:0/2989304849 shutdown_connections 2023-10-12T22:33:19.372 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.369+0000 7f37a97a8700 1 --2- 172.21.15.195:0/2989304849 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f378c02ff60 0x7f378c032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:19.372 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.369+0000 7f37a97a8700 1 --2- 172.21.15.195:0/2989304849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37a40f51a0 0x7f37a41053b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:19.372 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.369+0000 7f37a97a8700 1 -- 172.21.15.195:0/2989304849 >> 172.21.15.195:0/2989304849 conn(0x7f37a40f09a0 msgr2=0x7f37a40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:19.372 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.369+0000 7f37a97a8700 1 -- 172.21.15.195:0/2989304849 shutdown_connections 2023-10-12T22:33:19.372 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:19.369+0000 7f37a97a8700 1 -- 172.21.15.195:0/2989304849 wait complete. 2023-10-12T22:33:19.373 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:20.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:20 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:18.813312+0000 mgr.smithi131.efktma (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:20.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:20 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:19.369064+0000 mon.smithi131 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.195:0/2989304849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:20.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:20.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:21.906 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.905+0000 7fa3a1856700 1 -- 172.21.15.195:0/1575056278 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa39c0f51a0 msgr2=0x7fa39c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:21.907 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.905+0000 7fa3a1856700 1 --2- 172.21.15.195:0/1575056278 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa39c0f51a0 0x7fa39c0f55c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa390009fb0 tx=0x7fa390009b30).stop 2023-10-12T22:33:21.907 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.905+0000 7fa3a1856700 1 -- 172.21.15.195:0/1575056278 shutdown_connections 2023-10-12T22:33:21.907 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.905+0000 7fa3a1856700 1 --2- 172.21.15.195:0/1575056278 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa39c0f51a0 0x7fa39c0f55c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:21.907 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.905+0000 7fa3a1856700 1 -- 172.21.15.195:0/1575056278 >> 172.21.15.195:0/1575056278 conn(0x7fa39c0f09a0 msgr2=0x7fa39c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:21.908 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.905+0000 7fa3a1856700 1 -- 172.21.15.195:0/1575056278 shutdown_connections 2023-10-12T22:33:21.908 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.905+0000 7fa3a1856700 1 -- 172.21.15.195:0/1575056278 wait complete. 2023-10-12T22:33:21.908 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.905+0000 7fa3a1856700 1 Processor -- start 2023-10-12T22:33:21.908 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.905+0000 7fa3a1856700 1 -- start start 2023-10-12T22:33:21.910 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.906+0000 7fa3a1856700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa39c0f51a0 0x7fa39c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:21.910 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.906+0000 7fa3a1856700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c060240 con 0x7fa39c0f51a0 2023-10-12T22:33:21.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.906+0000 7fa3a0854700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa39c0f51a0 0x7fa39c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:21.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.906+0000 7fa3a0854700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa39c0f51a0 0x7fa39c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:34440/0 (socket says 172.21.15.195:34440) 2023-10-12T22:33:21.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.906+0000 7fa3a0854700 1 -- 172.21.15.195:0/1316518995 learned_addr learned my addr 172.21.15.195:0/1316518995 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:21.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.907+0000 7fa3a0854700 1 -- 172.21.15.195:0/1316518995 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa390005040 con 0x7fa39c0f51a0 2023-10-12T22:33:21.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.907+0000 7fa3a0854700 1 --2- 172.21.15.195:0/1316518995 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa39c0f51a0 0x7fa39c1091b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fa390013510 tx=0x7fa39000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:21.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.907+0000 7fa3997fa700 1 -- 172.21.15.195:0/1316518995 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa390009320 con 0x7fa39c0f51a0 2023-10-12T22:33:21.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.907+0000 7fa3a1856700 1 -- 172.21.15.195:0/1316518995 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39c060440 con 0x7fa39c0f51a0 2023-10-12T22:33:21.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.907+0000 7fa3997fa700 1 -- 172.21.15.195:0/1316518995 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa390013f00 con 0x7fa39c0f51a0 2023-10-12T22:33:21.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.907+0000 7fa3997fa700 1 -- 172.21.15.195:0/1316518995 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa390014dc0 con 0x7fa39c0f51a0 2023-10-12T22:33:21.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.907+0000 7fa3a1856700 1 -- 172.21.15.195:0/1316518995 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c109860 con 0x7fa39c0f51a0 2023-10-12T22:33:21.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.908+0000 7fa3997fa700 1 -- 172.21.15.195:0/1316518995 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa39001d3d0 con 0x7fa39c0f51a0 2023-10-12T22:33:21.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.908+0000 7fa3997fa700 1 --2- 172.21.15.195:0/1316518995 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa38c02ff60 0x7fa38c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:21.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.908+0000 7fa3997fa700 1 -- 172.21.15.195:0/1316518995 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa390041110 con 0x7fa39c0f51a0 2023-10-12T22:33:21.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.908+0000 7fa3a1856700 1 -- 172.21.15.195:0/1316518995 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c045ad0 con 0x7fa39c0f51a0 2023-10-12T22:33:21.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.908+0000 7fa39bfff700 1 --2- 172.21.15.195:0/1316518995 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa38c02ff60 0x7fa38c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:21.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.909+0000 7fa39bfff700 1 --2- 172.21.15.195:0/1316518995 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa38c02ff60 0x7fa38c032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa3880096a0 tx=0x7fa388006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:21.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:21.913+0000 7fa3997fa700 1 -- 172.21.15.195:0/1316518995 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa390012920 con 0x7fa39c0f51a0 2023-10-12T22:33:22.121 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:22.120+0000 7fa3a1856700 1 -- 172.21.15.195:0/1316518995 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa39c109f40 con 0x7fa39c0f51a0 2023-10-12T22:33:22.122 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:22.121+0000 7fa3997fa700 1 -- 172.21.15.195:0/1316518995 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa39002b020 con 0x7fa39c0f51a0 2023-10-12T22:33:22.122 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:22.122 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:22.123 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:22.122+0000 7fa3a1856700 1 -- 172.21.15.195:0/1316518995 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa38c02ff60 msgr2=0x7fa38c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:22.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:22.122+0000 7fa3a1856700 1 --2- 172.21.15.195:0/1316518995 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa38c02ff60 0x7fa38c032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa3880096a0 tx=0x7fa388006b40).stop 2023-10-12T22:33:22.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:22.122+0000 7fa3a1856700 1 -- 172.21.15.195:0/1316518995 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa39c0f51a0 msgr2=0x7fa39c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:22.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:22.122+0000 7fa3a1856700 1 --2- 172.21.15.195:0/1316518995 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa39c0f51a0 0x7fa39c1091b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fa390013510 tx=0x7fa39000b040).stop 2023-10-12T22:33:22.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:22.122+0000 7fa3a1856700 1 -- 172.21.15.195:0/1316518995 shutdown_connections 2023-10-12T22:33:22.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:22.122+0000 7fa3a1856700 1 --2- 172.21.15.195:0/1316518995 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa38c02ff60 0x7fa38c032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:22.125 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:22.122+0000 7fa3a1856700 1 --2- 172.21.15.195:0/1316518995 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa39c0f51a0 0x7fa39c1091b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:22.125 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:22.122+0000 7fa3a1856700 1 -- 172.21.15.195:0/1316518995 >> 172.21.15.195:0/1316518995 conn(0x7fa39c0f09a0 msgr2=0x7fa39c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:22.125 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:22.122+0000 7fa3a1856700 1 -- 172.21.15.195:0/1316518995 shutdown_connections 2023-10-12T22:33:22.125 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:22.122+0000 7fa3a1856700 1 -- 172.21.15.195:0/1316518995 wait complete. 2023-10-12T22:33:22.126 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:22.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:20.813674+0000 mgr.smithi131.efktma (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:23.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:22.122079+0000 mon.smithi131 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.195:0/1316518995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:23.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:22.931139+0000 mon.smithi131 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/mirror_snapshot_schedule"}]: dispatch 2023-10-12T22:33:23.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:22.938758+0000 mon.smithi131 (mon.0) 152 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/trash_purge_schedule"}]: dispatch 2023-10-12T22:33:23.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:23.515 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:24.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:22.814036+0000 mgr.smithi131.efktma (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:24.677 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.675+0000 7f6bc9625700 1 -- 172.21.15.195:0/3006686081 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bc40f6510 msgr2=0x7f6bc40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:24.677 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.675+0000 7f6bc9625700 1 --2- 172.21.15.195:0/3006686081 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bc40f6510 0x7f6bc40f6930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f6bb4009fb0 tx=0x7f6bb4009b30).stop 2023-10-12T22:33:24.678 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.675+0000 7f6bc9625700 1 -- 172.21.15.195:0/3006686081 shutdown_connections 2023-10-12T22:33:24.678 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.675+0000 7f6bc9625700 1 --2- 172.21.15.195:0/3006686081 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bc40f6510 0x7f6bc40f6930 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:24.678 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.675+0000 7f6bc9625700 1 -- 172.21.15.195:0/3006686081 >> 172.21.15.195:0/3006686081 conn(0x7f6bc40f1c70 msgr2=0x7f6bc40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:24.678 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.676+0000 7f6bc9625700 1 -- 172.21.15.195:0/3006686081 shutdown_connections 2023-10-12T22:33:24.678 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.676+0000 7f6bc9625700 1 -- 172.21.15.195:0/3006686081 wait complete. 2023-10-12T22:33:24.681 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.676+0000 7f6bc9625700 1 Processor -- start 2023-10-12T22:33:24.681 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.676+0000 7f6bc9625700 1 -- start start 2023-10-12T22:33:24.681 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.677+0000 7f6bc9625700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bc40f6510 0x7f6bc4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:24.681 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.677+0000 7f6bc9625700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc4106c20 con 0x7f6bc40f6510 2023-10-12T22:33:24.681 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.677+0000 7f6bc2ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bc40f6510 0x7f6bc4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:24.682 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.677+0000 7f6bc2ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bc40f6510 0x7f6bc4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:34468/0 (socket says 172.21.15.195:34468) 2023-10-12T22:33:24.682 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.677+0000 7f6bc2ffd700 1 -- 172.21.15.195:0/2166645015 learned_addr learned my addr 172.21.15.195:0/2166645015 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:24.682 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.677+0000 7f6bc2ffd700 1 -- 172.21.15.195:0/2166645015 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb4005040 con 0x7f6bc40f6510 2023-10-12T22:33:24.682 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.677+0000 7f6bc2ffd700 1 --2- 172.21.15.195:0/2166645015 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bc40f6510 0x7f6bc4106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f6bb4013510 tx=0x7f6bb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:24.682 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.677+0000 7f6bab7fe700 1 -- 172.21.15.195:0/2166645015 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bb4009320 con 0x7f6bc40f6510 2023-10-12T22:33:24.683 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.678+0000 7f6bc9625700 1 -- 172.21.15.195:0/2166645015 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc4106e20 con 0x7f6bc40f6510 2023-10-12T22:33:24.683 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.678+0000 7f6bab7fe700 1 -- 172.21.15.195:0/2166645015 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bb4013f00 con 0x7f6bc40f6510 2023-10-12T22:33:24.683 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.678+0000 7f6bab7fe700 1 -- 172.21.15.195:0/2166645015 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bb4014d90 con 0x7f6bc40f6510 2023-10-12T22:33:24.683 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.678+0000 7f6bc9625700 1 -- 172.21.15.195:0/2166645015 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc41072b0 con 0x7f6bc40f6510 2023-10-12T22:33:24.683 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.678+0000 7f6bab7fe700 1 -- 172.21.15.195:0/2166645015 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6bb40263a0 con 0x7f6bc40f6510 2023-10-12T22:33:24.684 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.678+0000 7f6bc9625700 1 -- 172.21.15.195:0/2166645015 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc4045ad0 con 0x7f6bc40f6510 2023-10-12T22:33:24.684 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.678+0000 7f6bab7fe700 1 --2- 172.21.15.195:0/2166645015 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f6bac02ff60 0x7f6bac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:24.684 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.679+0000 7f6bab7fe700 1 -- 172.21.15.195:0/2166645015 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6bb40413d0 con 0x7f6bc40f6510 2023-10-12T22:33:24.684 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.679+0000 7f6bc27fc700 1 --2- 172.21.15.195:0/2166645015 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f6bac02ff60 0x7f6bac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:24.684 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.679+0000 7f6bc27fc700 1 --2- 172.21.15.195:0/2166645015 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f6bac02ff60 0x7f6bac032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f6bb80096a0 tx=0x7f6bb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:24.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.683+0000 7f6bab7fe700 1 -- 172.21.15.195:0/2166645015 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bb40126f0 con 0x7f6bc40f6510 2023-10-12T22:33:24.890 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.889+0000 7f6bc9625700 1 -- 172.21.15.195:0/2166645015 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6bc40fbf60 con 0x7f6bc40f6510 2023-10-12T22:33:24.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.891+0000 7f6bab7fe700 1 -- 172.21.15.195:0/2166645015 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6bb4012330 con 0x7f6bc40f6510 2023-10-12T22:33:24.891 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:24.892 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:24.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.891+0000 7f6bc9625700 1 -- 172.21.15.195:0/2166645015 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f6bac02ff60 msgr2=0x7f6bac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:24.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.891+0000 7f6bc9625700 1 --2- 172.21.15.195:0/2166645015 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f6bac02ff60 0x7f6bac032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f6bb80096a0 tx=0x7f6bb8006b40).stop 2023-10-12T22:33:24.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.891+0000 7f6bc9625700 1 -- 172.21.15.195:0/2166645015 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bc40f6510 msgr2=0x7f6bc4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:24.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.891+0000 7f6bc9625700 1 --2- 172.21.15.195:0/2166645015 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bc40f6510 0x7f6bc4106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f6bb4013510 tx=0x7f6bb400b040).stop 2023-10-12T22:33:24.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.891+0000 7f6bc9625700 1 -- 172.21.15.195:0/2166645015 shutdown_connections 2023-10-12T22:33:24.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.891+0000 7f6bc9625700 1 --2- 172.21.15.195:0/2166645015 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f6bac02ff60 0x7f6bac032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:24.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.891+0000 7f6bc9625700 1 --2- 172.21.15.195:0/2166645015 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bc40f6510 0x7f6bc4106730 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:24.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.891+0000 7f6bc9625700 1 -- 172.21.15.195:0/2166645015 >> 172.21.15.195:0/2166645015 conn(0x7f6bc40f1c70 msgr2=0x7f6bc40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:24.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.891+0000 7f6bc9625700 1 -- 172.21.15.195:0/2166645015 shutdown_connections 2023-10-12T22:33:24.895 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:24.892+0000 7f6bc9625700 1 -- 172.21.15.195:0/2166645015 wait complete. 2023-10-12T22:33:24.895 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:25.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:25 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:24.891345+0000 mon.smithi131 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.195:0/2166645015' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:26.268 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:26.268 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:26.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:26 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:24.814391+0000 mgr.smithi131.efktma (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:27.453 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.450+0000 7f790359e700 1 -- 172.21.15.195:0/2889724098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79040f51a0 msgr2=0x7f79040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:27.454 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.450+0000 7f790359e700 1 --2- 172.21.15.195:0/2889724098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79040f51a0 0x7f79040f55c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f78f8009fb0 tx=0x7f78f8009b30).stop 2023-10-12T22:33:27.454 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.451+0000 7f790359e700 1 -- 172.21.15.195:0/2889724098 shutdown_connections 2023-10-12T22:33:27.454 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.451+0000 7f790359e700 1 --2- 172.21.15.195:0/2889724098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79040f51a0 0x7f79040f55c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:27.454 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.451+0000 7f790359e700 1 -- 172.21.15.195:0/2889724098 >> 172.21.15.195:0/2889724098 conn(0x7f79040f09a0 msgr2=0x7f79040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:27.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.451+0000 7f790359e700 1 -- 172.21.15.195:0/2889724098 shutdown_connections 2023-10-12T22:33:27.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.451+0000 7f790359e700 1 -- 172.21.15.195:0/2889724098 wait complete. 2023-10-12T22:33:27.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.451+0000 7f790359e700 1 Processor -- start 2023-10-12T22:33:27.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.451+0000 7f790359e700 1 -- start start 2023-10-12T22:33:27.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.451+0000 7f790359e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79040f51a0 0x7f79041091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:27.456 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.451+0000 7f790359e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7904060240 con 0x7f79040f51a0 2023-10-12T22:33:27.456 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.452+0000 7f790259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79040f51a0 0x7f79041091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:27.456 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.452+0000 7f790259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79040f51a0 0x7f79041091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:34490/0 (socket says 172.21.15.195:34490) 2023-10-12T22:33:27.456 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.452+0000 7f790259c700 1 -- 172.21.15.195:0/1815615593 learned_addr learned my addr 172.21.15.195:0/1815615593 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:27.456 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.452+0000 7f790259c700 1 -- 172.21.15.195:0/1815615593 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f8005040 con 0x7f79040f51a0 2023-10-12T22:33:27.457 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.452+0000 7f790259c700 1 --2- 172.21.15.195:0/1815615593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79040f51a0 0x7f79041091b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f78f8013510 tx=0x7f78f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:27.457 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.452+0000 7f78f2ffd700 1 -- 172.21.15.195:0/1815615593 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f8009320 con 0x7f79040f51a0 2023-10-12T22:33:27.457 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.452+0000 7f78f2ffd700 1 -- 172.21.15.195:0/1815615593 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78f8013f00 con 0x7f79040f51a0 2023-10-12T22:33:27.457 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.452+0000 7f790359e700 1 -- 172.21.15.195:0/1815615593 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7904060440 con 0x7f79040f51a0 2023-10-12T22:33:27.457 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.452+0000 7f78f2ffd700 1 -- 172.21.15.195:0/1815615593 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f8014dc0 con 0x7f79040f51a0 2023-10-12T22:33:27.458 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.452+0000 7f790359e700 1 -- 172.21.15.195:0/1815615593 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7904109860 con 0x7f79040f51a0 2023-10-12T22:33:27.459 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.453+0000 7f78f2ffd700 1 -- 172.21.15.195:0/1815615593 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78f80095f0 con 0x7f79040f51a0 2023-10-12T22:33:27.459 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.453+0000 7f790359e700 1 -- 172.21.15.195:0/1815615593 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7904045ad0 con 0x7f79040f51a0 2023-10-12T22:33:27.459 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.453+0000 7f78f2ffd700 1 --2- 172.21.15.195:0/1815615593 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f78ec02ff60 0x7f78ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:27.459 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.453+0000 7f78f2ffd700 1 -- 172.21.15.195:0/1815615593 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78f80410c0 con 0x7f79040f51a0 2023-10-12T22:33:27.459 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.453+0000 7f7901d9b700 1 --2- 172.21.15.195:0/1815615593 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f78ec02ff60 0x7f78ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:27.460 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.454+0000 7f7901d9b700 1 --2- 172.21.15.195:0/1815615593 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f78ec02ff60 0x7f78ec032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f78f40096a0 tx=0x7f78f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:27.460 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.458+0000 7f78f2ffd700 1 -- 172.21.15.195:0/1815615593 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78f8014420 con 0x7f79040f51a0 2023-10-12T22:33:27.664 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.663+0000 7f790359e700 1 -- 172.21.15.195:0/1815615593 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f790410a080 con 0x7f79040f51a0 2023-10-12T22:33:27.665 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.664+0000 7f78f2ffd700 1 -- 172.21.15.195:0/1815615593 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f78f8010230 con 0x7f79040f51a0 2023-10-12T22:33:27.666 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:27.666 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:27.667 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.665+0000 7f790359e700 1 -- 172.21.15.195:0/1815615593 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f78ec02ff60 msgr2=0x7f78ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:27.667 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.665+0000 7f790359e700 1 --2- 172.21.15.195:0/1815615593 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f78ec02ff60 0x7f78ec032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f78f40096a0 tx=0x7f78f4006b40).stop 2023-10-12T22:33:27.667 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.665+0000 7f790359e700 1 -- 172.21.15.195:0/1815615593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79040f51a0 msgr2=0x7f79041091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:27.667 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.665+0000 7f790359e700 1 --2- 172.21.15.195:0/1815615593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79040f51a0 0x7f79041091b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f78f8013510 tx=0x7f78f800b040).stop 2023-10-12T22:33:27.668 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.665+0000 7f790359e700 1 -- 172.21.15.195:0/1815615593 shutdown_connections 2023-10-12T22:33:27.668 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.666+0000 7f790359e700 1 --2- 172.21.15.195:0/1815615593 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f78ec02ff60 0x7f78ec032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:27.668 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.666+0000 7f790359e700 1 --2- 172.21.15.195:0/1815615593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79040f51a0 0x7f79041091b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:27.668 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.666+0000 7f790359e700 1 -- 172.21.15.195:0/1815615593 >> 172.21.15.195:0/1815615593 conn(0x7f79040f09a0 msgr2=0x7f79040f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:27.668 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.666+0000 7f790359e700 1 -- 172.21.15.195:0/1815615593 shutdown_connections 2023-10-12T22:33:27.669 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:27.666+0000 7f790359e700 1 -- 172.21.15.195:0/1815615593 wait complete. 2023-10-12T22:33:27.669 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:28.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:28 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:26.814689+0000 mgr.smithi131.efktma (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:28.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:28 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:27.665434+0000 mon.smithi131 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.195:0/1815615593' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:29.086 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:29.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:30.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.195+0000 7f97a25bc700 1 -- 172.21.15.195:0/1476657336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c0f5410 msgr2=0x7f979c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:30.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.195+0000 7f97a25bc700 1 --2- 172.21.15.195:0/1476657336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c0f5410 0x7f979c0f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9790009fb0 tx=0x7f9790009b30).stop 2023-10-12T22:33:30.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.195+0000 7f97a25bc700 1 -- 172.21.15.195:0/1476657336 shutdown_connections 2023-10-12T22:33:30.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.195+0000 7f97a25bc700 1 --2- 172.21.15.195:0/1476657336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c0f5410 0x7f979c0f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:30.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.195+0000 7f97a25bc700 1 -- 172.21.15.195:0/1476657336 >> 172.21.15.195:0/1476657336 conn(0x7f979c0f0b70 msgr2=0x7f979c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:30.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.195+0000 7f97a25bc700 1 -- 172.21.15.195:0/1476657336 shutdown_connections 2023-10-12T22:33:30.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.195+0000 7f97a25bc700 1 -- 172.21.15.195:0/1476657336 wait complete. 2023-10-12T22:33:30.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.196+0000 7f97a25bc700 1 Processor -- start 2023-10-12T22:33:30.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.196+0000 7f97a25bc700 1 -- start start 2023-10-12T22:33:30.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.196+0000 7f97a25bc700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c0f5410 0x7f979c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:30.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.196+0000 7f97a25bc700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c105b20 con 0x7f979c0f5410 2023-10-12T22:33:30.201 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.196+0000 7f979bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c0f5410 0x7f979c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:30.201 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.196+0000 7f979bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c0f5410 0x7f979c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:44464/0 (socket says 172.21.15.195:44464) 2023-10-12T22:33:30.201 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.196+0000 7f979bfff700 1 -- 172.21.15.195:0/1103000804 learned_addr learned my addr 172.21.15.195:0/1103000804 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:30.201 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.197+0000 7f979bfff700 1 -- 172.21.15.195:0/1103000804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9790005040 con 0x7f979c0f5410 2023-10-12T22:33:30.202 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.197+0000 7f979bfff700 1 --2- 172.21.15.195:0/1103000804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c0f5410 0x7f979c105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9790013510 tx=0x7f979000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:30.202 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.197+0000 7f9798ff9700 1 -- 172.21.15.195:0/1103000804 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9790009320 con 0x7f979c0f5410 2023-10-12T22:33:30.202 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.197+0000 7f9798ff9700 1 -- 172.21.15.195:0/1103000804 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9790013f00 con 0x7f979c0f5410 2023-10-12T22:33:30.202 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.197+0000 7f97a25bc700 1 -- 172.21.15.195:0/1103000804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979c105d20 con 0x7f979c0f5410 2023-10-12T22:33:30.202 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.197+0000 7f9798ff9700 1 -- 172.21.15.195:0/1103000804 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9790014d90 con 0x7f979c0f5410 2023-10-12T22:33:30.203 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.197+0000 7f97a25bc700 1 -- 172.21.15.195:0/1103000804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f979c1061b0 con 0x7f979c0f5410 2023-10-12T22:33:30.203 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.198+0000 7f9798ff9700 1 -- 172.21.15.195:0/1103000804 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97900095f0 con 0x7f979c0f5410 2023-10-12T22:33:30.203 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.198+0000 7f9798ff9700 1 --2- 172.21.15.195:0/1103000804 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f978402ff60 0x7f9784032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:30.203 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.198+0000 7f9798ff9700 1 -- 172.21.15.195:0/1103000804 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9790041000 con 0x7f979c0f5410 2023-10-12T22:33:30.204 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.198+0000 7f97a25bc700 1 -- 172.21.15.195:0/1103000804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979c059100 con 0x7f979c0f5410 2023-10-12T22:33:30.204 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.198+0000 7f979b7fe700 1 --2- 172.21.15.195:0/1103000804 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f978402ff60 0x7f9784032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:30.204 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.199+0000 7f979b7fe700 1 --2- 172.21.15.195:0/1103000804 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f978402ff60 0x7f9784032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f978c0096a0 tx=0x7f978c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:30.204 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.203+0000 7f9798ff9700 1 -- 172.21.15.195:0/1103000804 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9790014420 con 0x7f979c0f5410 2023-10-12T22:33:30.411 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.410+0000 7f97a25bc700 1 -- 172.21.15.195:0/1103000804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f979c0fae60 con 0x7f979c0f5410 2023-10-12T22:33:30.412 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.411+0000 7f9798ff9700 1 -- 172.21.15.195:0/1103000804 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f979002b020 con 0x7f979c0f5410 2023-10-12T22:33:30.412 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:30.413 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:30.413 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.412+0000 7f97a25bc700 1 -- 172.21.15.195:0/1103000804 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f978402ff60 msgr2=0x7f9784032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:30.414 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.412+0000 7f97a25bc700 1 --2- 172.21.15.195:0/1103000804 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f978402ff60 0x7f9784032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f978c0096a0 tx=0x7f978c006b40).stop 2023-10-12T22:33:30.414 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.412+0000 7f97a25bc700 1 -- 172.21.15.195:0/1103000804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c0f5410 msgr2=0x7f979c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:30.414 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.412+0000 7f97a25bc700 1 --2- 172.21.15.195:0/1103000804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c0f5410 0x7f979c105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9790013510 tx=0x7f979000b040).stop 2023-10-12T22:33:30.414 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.412+0000 7f97a25bc700 1 -- 172.21.15.195:0/1103000804 shutdown_connections 2023-10-12T22:33:30.415 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.412+0000 7f97a25bc700 1 --2- 172.21.15.195:0/1103000804 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f978402ff60 0x7f9784032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:30.415 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.412+0000 7f97a25bc700 1 --2- 172.21.15.195:0/1103000804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c0f5410 0x7f979c105630 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:30.415 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.412+0000 7f97a25bc700 1 -- 172.21.15.195:0/1103000804 >> 172.21.15.195:0/1103000804 conn(0x7f979c0f0b70 msgr2=0x7f979c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:30.415 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.412+0000 7f97a25bc700 1 -- 172.21.15.195:0/1103000804 shutdown_connections 2023-10-12T22:33:30.415 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:30.412+0000 7f97a25bc700 1 -- 172.21.15.195:0/1103000804 wait complete. 2023-10-12T22:33:30.416 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:30.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:30 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:28.815022+0000 mgr.smithi131.efktma (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:31.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:31 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:30.412115+0000 mon.smithi131 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.195:0/1103000804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:31.823 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:31.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:32.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:30.815392+0000 mgr.smithi131.efktma (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:32.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.983+0000 7f3b0ea85700 1 -- 172.21.15.195:0/1229629907 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b080f5410 msgr2=0x7f3b080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:32.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.983+0000 7f3b0ea85700 1 --2- 172.21.15.195:0/1229629907 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b080f5410 0x7f3b080f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f3afc009fb0 tx=0x7f3afc009b30).stop 2023-10-12T22:33:32.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.983+0000 7f3b0ea85700 1 -- 172.21.15.195:0/1229629907 shutdown_connections 2023-10-12T22:33:32.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.983+0000 7f3b0ea85700 1 --2- 172.21.15.195:0/1229629907 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b080f5410 0x7f3b080f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:32.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.983+0000 7f3b0ea85700 1 -- 172.21.15.195:0/1229629907 >> 172.21.15.195:0/1229629907 conn(0x7f3b080f0b70 msgr2=0x7f3b080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:32.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.983+0000 7f3b0ea85700 1 -- 172.21.15.195:0/1229629907 shutdown_connections 2023-10-12T22:33:32.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.983+0000 7f3b0ea85700 1 -- 172.21.15.195:0/1229629907 wait complete. 2023-10-12T22:33:32.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.984+0000 7f3b0ea85700 1 Processor -- start 2023-10-12T22:33:32.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.984+0000 7f3b0ea85700 1 -- start start 2023-10-12T22:33:32.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.984+0000 7f3b0ea85700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b080f5410 0x7f3b08106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:32.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.984+0000 7f3b0ea85700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b08106c10 con 0x7f3b080f5410 2023-10-12T22:33:32.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.984+0000 7f3b0c821700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b080f5410 0x7f3b08106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:32.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.984+0000 7f3b0c821700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b080f5410 0x7f3b08106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:44468/0 (socket says 172.21.15.195:44468) 2023-10-12T22:33:32.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.984+0000 7f3b0c821700 1 -- 172.21.15.195:0/652359193 learned_addr learned my addr 172.21.15.195:0/652359193 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:32.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.985+0000 7f3b0c821700 1 -- 172.21.15.195:0/652359193 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc005040 con 0x7f3b080f5410 2023-10-12T22:33:32.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.985+0000 7f3b0c821700 1 --2- 172.21.15.195:0/652359193 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b080f5410 0x7f3b08106720 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3afc013510 tx=0x7f3afc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:32.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.985+0000 7f3b057fa700 1 -- 172.21.15.195:0/652359193 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3afc009320 con 0x7f3b080f5410 2023-10-12T22:33:32.989 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.985+0000 7f3b0ea85700 1 -- 172.21.15.195:0/652359193 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b08106e10 con 0x7f3b080f5410 2023-10-12T22:33:32.989 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.985+0000 7f3b057fa700 1 -- 172.21.15.195:0/652359193 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3afc013f00 con 0x7f3b080f5410 2023-10-12T22:33:32.989 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.985+0000 7f3b057fa700 1 -- 172.21.15.195:0/652359193 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3afc014d90 con 0x7f3b080f5410 2023-10-12T22:33:32.990 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.985+0000 7f3b0ea85700 1 -- 172.21.15.195:0/652359193 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b081072a0 con 0x7f3b080f5410 2023-10-12T22:33:32.991 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.986+0000 7f3b057fa700 1 -- 172.21.15.195:0/652359193 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3afc0263a0 con 0x7f3b080f5410 2023-10-12T22:33:32.991 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.986+0000 7f3b057fa700 1 --2- 172.21.15.195:0/652359193 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3af802ff60 0x7f3af8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:32.991 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.986+0000 7f3b057fa700 1 -- 172.21.15.195:0/652359193 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3afc041000 con 0x7f3b080f5410 2023-10-12T22:33:32.992 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.986+0000 7f3b0ea85700 1 -- 172.21.15.195:0/652359193 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b08059100 con 0x7f3b080f5410 2023-10-12T22:33:32.992 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.986+0000 7f3b07fff700 1 --2- 172.21.15.195:0/652359193 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3af802ff60 0x7f3af8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:32.992 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.986+0000 7f3b07fff700 1 --2- 172.21.15.195:0/652359193 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3af802ff60 0x7f3af8032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f3af40096a0 tx=0x7f3af4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:32.992 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:32.991+0000 7f3b057fa700 1 -- 172.21.15.195:0/652359193 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3afc012920 con 0x7f3b080f5410 2023-10-12T22:33:33.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:33.196+0000 7f3b0ea85700 1 -- 172.21.15.195:0/652359193 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b080fae60 con 0x7f3b080f5410 2023-10-12T22:33:33.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:33.197+0000 7f3b057fa700 1 -- 172.21.15.195:0/652359193 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3afc02b020 con 0x7f3b080f5410 2023-10-12T22:33:33.198 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:33.198 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:33.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:33.199+0000 7f3b0ea85700 1 -- 172.21.15.195:0/652359193 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3af802ff60 msgr2=0x7f3af8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:33.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:33.199+0000 7f3b0ea85700 1 --2- 172.21.15.195:0/652359193 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3af802ff60 0x7f3af8032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f3af40096a0 tx=0x7f3af4006b40).stop 2023-10-12T22:33:33.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:33.199+0000 7f3b0ea85700 1 -- 172.21.15.195:0/652359193 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b080f5410 msgr2=0x7f3b08106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:33.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:33.199+0000 7f3b0ea85700 1 --2- 172.21.15.195:0/652359193 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b080f5410 0x7f3b08106720 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3afc013510 tx=0x7f3afc00b040).stop 2023-10-12T22:33:33.200 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:33.199+0000 7f3b0ea85700 1 -- 172.21.15.195:0/652359193 shutdown_connections 2023-10-12T22:33:33.200 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:33.199+0000 7f3b0ea85700 1 --2- 172.21.15.195:0/652359193 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3af802ff60 0x7f3af8032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:33.200 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:33.199+0000 7f3b0ea85700 1 --2- 172.21.15.195:0/652359193 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3b080f5410 0x7f3b08106720 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:33.200 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:33.199+0000 7f3b0ea85700 1 -- 172.21.15.195:0/652359193 >> 172.21.15.195:0/652359193 conn(0x7f3b080f0b70 msgr2=0x7f3b0805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:33.201 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:33.199+0000 7f3b0ea85700 1 -- 172.21.15.195:0/652359193 shutdown_connections 2023-10-12T22:33:33.201 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:33.199+0000 7f3b0ea85700 1 -- 172.21.15.195:0/652359193 wait complete. 2023-10-12T22:33:33.201 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:34.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:32.815759+0000 mgr.smithi131.efktma (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:34.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:33.197420+0000 mon.smithi131 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.195:0/652359193' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:34.590 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:34.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:35.783 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.781+0000 7fed5281e700 1 -- 172.21.15.195:0/2814581141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed4c0f6510 msgr2=0x7fed4c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:35.784 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.781+0000 7fed5281e700 1 --2- 172.21.15.195:0/2814581141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed4c0f6510 0x7fed4c0f6930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fed3c009fb0 tx=0x7fed3c009b30).stop 2023-10-12T22:33:35.784 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.781+0000 7fed5281e700 1 -- 172.21.15.195:0/2814581141 shutdown_connections 2023-10-12T22:33:35.784 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.781+0000 7fed5281e700 1 --2- 172.21.15.195:0/2814581141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed4c0f6510 0x7fed4c0f6930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:35.784 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.781+0000 7fed5281e700 1 -- 172.21.15.195:0/2814581141 >> 172.21.15.195:0/2814581141 conn(0x7fed4c0f1c70 msgr2=0x7fed4c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:35.784 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.781+0000 7fed5281e700 1 -- 172.21.15.195:0/2814581141 shutdown_connections 2023-10-12T22:33:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.781+0000 7fed5281e700 1 -- 172.21.15.195:0/2814581141 wait complete. 2023-10-12T22:33:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.782+0000 7fed5281e700 1 Processor -- start 2023-10-12T22:33:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.782+0000 7fed5281e700 1 -- start start 2023-10-12T22:33:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.782+0000 7fed5281e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed4c0f6510 0x7fed4c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.782+0000 7fed5281e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c069e20 con 0x7fed4c0f6510 2023-10-12T22:33:35.786 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.782+0000 7fed4bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed4c0f6510 0x7fed4c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:35.786 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.782+0000 7fed4bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed4c0f6510 0x7fed4c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:44486/0 (socket says 172.21.15.195:44486) 2023-10-12T22:33:35.786 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.783+0000 7fed4bfff700 1 -- 172.21.15.195:0/942069586 learned_addr learned my addr 172.21.15.195:0/942069586 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:35.786 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.783+0000 7fed4bfff700 1 -- 172.21.15.195:0/942069586 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c005040 con 0x7fed4c0f6510 2023-10-12T22:33:35.787 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.783+0000 7fed4bfff700 1 --2- 172.21.15.195:0/942069586 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed4c0f6510 0x7fed4c06b9a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fed3c013510 tx=0x7fed3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:35.787 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.783+0000 7fed48ff9700 1 -- 172.21.15.195:0/942069586 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed3c009320 con 0x7fed4c0f6510 2023-10-12T22:33:35.787 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.783+0000 7fed48ff9700 1 -- 172.21.15.195:0/942069586 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed3c013f00 con 0x7fed4c0f6510 2023-10-12T22:33:35.787 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.783+0000 7fed48ff9700 1 -- 172.21.15.195:0/942069586 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed3c014d90 con 0x7fed4c0f6510 2023-10-12T22:33:35.787 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.783+0000 7fed5281e700 1 -- 172.21.15.195:0/942069586 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4c069fc0 con 0x7fed4c0f6510 2023-10-12T22:33:35.787 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.783+0000 7fed5281e700 1 -- 172.21.15.195:0/942069586 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4c06a450 con 0x7fed4c0f6510 2023-10-12T22:33:35.788 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.784+0000 7fed48ff9700 1 -- 172.21.15.195:0/942069586 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed3c014420 con 0x7fed4c0f6510 2023-10-12T22:33:35.789 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.784+0000 7fed48ff9700 1 --2- 172.21.15.195:0/942069586 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fed34030000 0x7fed340324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:35.789 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.784+0000 7fed5281e700 1 -- 172.21.15.195:0/942069586 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4c059100 con 0x7fed4c0f6510 2023-10-12T22:33:35.789 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.784+0000 7fed48ff9700 1 -- 172.21.15.195:0/942069586 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed3c041090 con 0x7fed4c0f6510 2023-10-12T22:33:35.789 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.784+0000 7fed4b7fe700 1 --2- 172.21.15.195:0/942069586 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fed34030000 0x7fed340324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:35.789 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.785+0000 7fed4b7fe700 1 --2- 172.21.15.195:0/942069586 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fed34030000 0x7fed340324c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fed400096a0 tx=0x7fed40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:35.790 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.789+0000 7fed48ff9700 1 -- 172.21.15.195:0/942069586 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed3c0095f0 con 0x7fed4c0f6510 2023-10-12T22:33:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.998+0000 7fed5281e700 1 -- 172.21.15.195:0/942069586 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed4c0fbf60 con 0x7fed4c0f6510 2023-10-12T22:33:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:35.999+0000 7fed48ff9700 1 -- 172.21.15.195:0/942069586 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fed3c01db70 con 0x7fed4c0f6510 2023-10-12T22:33:36.002 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:36.003 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:36.000+0000 7fed5281e700 1 -- 172.21.15.195:0/942069586 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fed34030000 msgr2=0x7fed340324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:36.000+0000 7fed5281e700 1 --2- 172.21.15.195:0/942069586 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fed34030000 0x7fed340324c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fed400096a0 tx=0x7fed40006b40).stop 2023-10-12T22:33:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:36.000+0000 7fed5281e700 1 -- 172.21.15.195:0/942069586 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed4c0f6510 msgr2=0x7fed4c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:36.000+0000 7fed5281e700 1 --2- 172.21.15.195:0/942069586 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed4c0f6510 0x7fed4c06b9a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fed3c013510 tx=0x7fed3c00b040).stop 2023-10-12T22:33:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:36.000+0000 7fed5281e700 1 -- 172.21.15.195:0/942069586 shutdown_connections 2023-10-12T22:33:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:36.000+0000 7fed5281e700 1 --2- 172.21.15.195:0/942069586 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fed34030000 0x7fed340324c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:36.000+0000 7fed5281e700 1 --2- 172.21.15.195:0/942069586 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed4c0f6510 0x7fed4c06b9a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:36.000+0000 7fed5281e700 1 -- 172.21.15.195:0/942069586 >> 172.21.15.195:0/942069586 conn(0x7fed4c0f1c70 msgr2=0x7fed4c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:36.001+0000 7fed5281e700 1 -- 172.21.15.195:0/942069586 shutdown_connections 2023-10-12T22:33:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:36.001+0000 7fed5281e700 1 -- 172.21.15.195:0/942069586 wait complete. 2023-10-12T22:33:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:36.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:36 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:34.816111+0000 mgr.smithi131.efktma (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:36.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:36 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:35.999202+0000 mon.smithi131 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.195:0/942069586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:37.340 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:37.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:38.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:38 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:36.816512+0000 mgr.smithi131.efktma (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:38.537 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.536+0000 7f673d94d700 1 -- 172.21.15.195:0/3475104637 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67380f4860 msgr2=0x7f67380f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:38.537 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.536+0000 7f673d94d700 1 --2- 172.21.15.195:0/3475104637 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67380f4860 0x7f67380f4c80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f672c009fb0 tx=0x7f672c009b30).stop 2023-10-12T22:33:38.537 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.536+0000 7f673d94d700 1 -- 172.21.15.195:0/3475104637 shutdown_connections 2023-10-12T22:33:38.537 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.536+0000 7f673d94d700 1 --2- 172.21.15.195:0/3475104637 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67380f4860 0x7f67380f4c80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:38.538 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.536+0000 7f673d94d700 1 -- 172.21.15.195:0/3475104637 >> 172.21.15.195:0/3475104637 conn(0x7f67380f02e0 msgr2=0x7f67380f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:38.538 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.536+0000 7f673d94d700 1 -- 172.21.15.195:0/3475104637 shutdown_connections 2023-10-12T22:33:38.538 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.536+0000 7f673d94d700 1 -- 172.21.15.195:0/3475104637 wait complete. 2023-10-12T22:33:38.540 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.537+0000 7f673d94d700 1 Processor -- start 2023-10-12T22:33:38.541 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.537+0000 7f673d94d700 1 -- start start 2023-10-12T22:33:38.541 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.537+0000 7f673d94d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67380f4860 0x7f67380f7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:38.541 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.537+0000 7f673d94d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67380f58d0 con 0x7f67380f4860 2023-10-12T22:33:38.541 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.537+0000 7f6736ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67380f4860 0x7f67380f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:38.541 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.538+0000 7f6736ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67380f4860 0x7f67380f7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:50400/0 (socket says 172.21.15.195:50400) 2023-10-12T22:33:38.542 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.538+0000 7f6736ffd700 1 -- 172.21.15.195:0/1315552213 learned_addr learned my addr 172.21.15.195:0/1315552213 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:38.542 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.538+0000 7f6736ffd700 1 -- 172.21.15.195:0/1315552213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c005040 con 0x7f67380f4860 2023-10-12T22:33:38.542 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.538+0000 7f6736ffd700 1 --2- 172.21.15.195:0/1315552213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67380f4860 0x7f67380f7450 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f672c009830 tx=0x7f672c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:38.542 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.538+0000 7f671f7fe700 1 -- 172.21.15.195:0/1315552213 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672c009320 con 0x7f67380f4860 2023-10-12T22:33:38.543 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.538+0000 7f671f7fe700 1 -- 172.21.15.195:0/1315552213 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f672c0136c0 con 0x7f67380f4860 2023-10-12T22:33:38.543 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.538+0000 7f673d94d700 1 -- 172.21.15.195:0/1315552213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67380f5a70 con 0x7f67380f4860 2023-10-12T22:33:38.543 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.538+0000 7f671f7fe700 1 -- 172.21.15.195:0/1315552213 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672c01ee90 con 0x7f67380f4860 2023-10-12T22:33:38.543 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.538+0000 7f673d94d700 1 -- 172.21.15.195:0/1315552213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67380f5f00 con 0x7f67380f4860 2023-10-12T22:33:38.543 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.539+0000 7f671f7fe700 1 -- 172.21.15.195:0/1315552213 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f672c01e420 con 0x7f67380f4860 2023-10-12T22:33:38.544 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.539+0000 7f673d94d700 1 -- 172.21.15.195:0/1315552213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6738059100 con 0x7f67380f4860 2023-10-12T22:33:38.544 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.539+0000 7f671f7fe700 1 --2- 172.21.15.195:0/1315552213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f672002ff10 0x7f67200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:38.544 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.539+0000 7f671f7fe700 1 -- 172.21.15.195:0/1315552213 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f672c041020 con 0x7f67380f4860 2023-10-12T22:33:38.544 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.539+0000 7f67367fc700 1 --2- 172.21.15.195:0/1315552213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f672002ff10 0x7f67200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:38.544 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.540+0000 7f67367fc700 1 --2- 172.21.15.195:0/1315552213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f672002ff10 0x7f67200323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f67280096a0 tx=0x7f6728006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:38.545 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.544+0000 7f671f7fe700 1 -- 172.21.15.195:0/1315552213 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f672c0095f0 con 0x7f67380f4860 2023-10-12T22:33:38.750 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.750+0000 7f673d94d700 1 -- 172.21.15.195:0/1315552213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6738045ad0 con 0x7f67380f4860 2023-10-12T22:33:38.751 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.751+0000 7f671f7fe700 1 -- 172.21.15.195:0/1315552213 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f672c02c2d0 con 0x7f67380f4860 2023-10-12T22:33:38.751 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:38.752 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:38.753 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.753+0000 7f673d94d700 1 -- 172.21.15.195:0/1315552213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f672002ff10 msgr2=0x7f67200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:38.754 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.753+0000 7f673d94d700 1 --2- 172.21.15.195:0/1315552213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f672002ff10 0x7f67200323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f67280096a0 tx=0x7f6728006b40).stop 2023-10-12T22:33:38.754 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.753+0000 7f673d94d700 1 -- 172.21.15.195:0/1315552213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67380f4860 msgr2=0x7f67380f7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:38.754 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.753+0000 7f673d94d700 1 --2- 172.21.15.195:0/1315552213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67380f4860 0x7f67380f7450 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f672c009830 tx=0x7f672c00b040).stop 2023-10-12T22:33:38.754 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.753+0000 7f673d94d700 1 -- 172.21.15.195:0/1315552213 shutdown_connections 2023-10-12T22:33:38.754 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.753+0000 7f673d94d700 1 --2- 172.21.15.195:0/1315552213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f672002ff10 0x7f67200323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:38.755 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.753+0000 7f673d94d700 1 --2- 172.21.15.195:0/1315552213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67380f4860 0x7f67380f7450 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:38.755 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.753+0000 7f673d94d700 1 -- 172.21.15.195:0/1315552213 >> 172.21.15.195:0/1315552213 conn(0x7f67380f02e0 msgr2=0x7f67380f0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:38.755 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.753+0000 7f673d94d700 1 -- 172.21.15.195:0/1315552213 shutdown_connections 2023-10-12T22:33:38.755 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:38.753+0000 7f673d94d700 1 -- 172.21.15.195:0/1315552213 wait complete. 2023-10-12T22:33:38.756 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:39.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:39 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:38.751392+0000 mon.smithi131 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.195:0/1315552213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:40.134 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:40.134 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:40.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:40 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:38.816803+0000 mgr.smithi131.efktma (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:41.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.260+0000 7fd317fff700 1 -- 172.21.15.195:0/2957598532 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3180f51a0 msgr2=0x7fd3180f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:41.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.260+0000 7fd317fff700 1 --2- 172.21.15.195:0/2957598532 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3180f51a0 0x7fd3180f55c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd308009fb0 tx=0x7fd308009b30).stop 2023-10-12T22:33:41.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.260+0000 7fd317fff700 1 -- 172.21.15.195:0/2957598532 shutdown_connections 2023-10-12T22:33:41.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.260+0000 7fd317fff700 1 --2- 172.21.15.195:0/2957598532 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3180f51a0 0x7fd3180f55c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:41.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.260+0000 7fd317fff700 1 -- 172.21.15.195:0/2957598532 >> 172.21.15.195:0/2957598532 conn(0x7fd3180f09a0 msgr2=0x7fd3180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:41.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.260+0000 7fd317fff700 1 -- 172.21.15.195:0/2957598532 shutdown_connections 2023-10-12T22:33:41.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.260+0000 7fd317fff700 1 -- 172.21.15.195:0/2957598532 wait complete. 2023-10-12T22:33:41.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.260+0000 7fd317fff700 1 Processor -- start 2023-10-12T22:33:41.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.260+0000 7fd317fff700 1 -- start start 2023-10-12T22:33:41.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.261+0000 7fd317fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3180f51a0 0x7fd3181064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:41.263 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.261+0000 7fd317fff700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3181069b0 con 0x7fd3180f51a0 2023-10-12T22:33:41.264 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.261+0000 7fd316ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3180f51a0 0x7fd3181064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:41.264 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.261+0000 7fd316ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3180f51a0 0x7fd3181064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:50418/0 (socket says 172.21.15.195:50418) 2023-10-12T22:33:41.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.261+0000 7fd316ffd700 1 -- 172.21.15.195:0/1438009714 learned_addr learned my addr 172.21.15.195:0/1438009714 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:41.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.261+0000 7fd316ffd700 1 -- 172.21.15.195:0/1438009714 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd308005040 con 0x7fd3180f51a0 2023-10-12T22:33:41.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.261+0000 7fd316ffd700 1 --2- 172.21.15.195:0/1438009714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3180f51a0 0x7fd3181064c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fd308013510 tx=0x7fd30800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:41.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.261+0000 7fd2ff7fe700 1 -- 172.21.15.195:0/1438009714 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd308009320 con 0x7fd3180f51a0 2023-10-12T22:33:41.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.261+0000 7fd2ff7fe700 1 -- 172.21.15.195:0/1438009714 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd308013f00 con 0x7fd3180f51a0 2023-10-12T22:33:41.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.261+0000 7fd317fff700 1 -- 172.21.15.195:0/1438009714 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd318106bb0 con 0x7fd3180f51a0 2023-10-12T22:33:41.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.261+0000 7fd2ff7fe700 1 -- 172.21.15.195:0/1438009714 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd308014dc0 con 0x7fd3180f51a0 2023-10-12T22:33:41.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.261+0000 7fd317fff700 1 -- 172.21.15.195:0/1438009714 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd318107040 con 0x7fd3180f51a0 2023-10-12T22:33:41.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.262+0000 7fd2ff7fe700 1 -- 172.21.15.195:0/1438009714 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3080095f0 con 0x7fd3180f51a0 2023-10-12T22:33:41.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.262+0000 7fd317fff700 1 -- 172.21.15.195:0/1438009714 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd318045ad0 con 0x7fd3180f51a0 2023-10-12T22:33:41.267 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.262+0000 7fd2ff7fe700 1 --2- 172.21.15.195:0/1438009714 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd30002ff60 0x7fd300032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:41.267 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.262+0000 7fd2ff7fe700 1 -- 172.21.15.195:0/1438009714 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3080410e0 con 0x7fd3180f51a0 2023-10-12T22:33:41.267 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.263+0000 7fd3167fc700 1 --2- 172.21.15.195:0/1438009714 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd30002ff60 0x7fd300032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:41.267 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.263+0000 7fd3167fc700 1 --2- 172.21.15.195:0/1438009714 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd30002ff60 0x7fd300032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fd30c0096a0 tx=0x7fd30c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:41.268 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.267+0000 7fd2ff7fe700 1 -- 172.21.15.195:0/1438009714 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd308014420 con 0x7fd3180f51a0 2023-10-12T22:33:41.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.474+0000 7fd317fff700 1 -- 172.21.15.195:0/1438009714 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd318107e90 con 0x7fd3180f51a0 2023-10-12T22:33:41.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.475+0000 7fd2ff7fe700 1 -- 172.21.15.195:0/1438009714 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd30801c920 con 0x7fd3180f51a0 2023-10-12T22:33:41.476 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:41.476 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:41.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.477+0000 7fd317fff700 1 -- 172.21.15.195:0/1438009714 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd30002ff60 msgr2=0x7fd300032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:41.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.477+0000 7fd317fff700 1 --2- 172.21.15.195:0/1438009714 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd30002ff60 0x7fd300032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fd30c0096a0 tx=0x7fd30c006b40).stop 2023-10-12T22:33:41.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.477+0000 7fd317fff700 1 -- 172.21.15.195:0/1438009714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3180f51a0 msgr2=0x7fd3181064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:41.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.477+0000 7fd317fff700 1 --2- 172.21.15.195:0/1438009714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3180f51a0 0x7fd3181064c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fd308013510 tx=0x7fd30800b040).stop 2023-10-12T22:33:41.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.477+0000 7fd317fff700 1 -- 172.21.15.195:0/1438009714 shutdown_connections 2023-10-12T22:33:41.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.477+0000 7fd317fff700 1 --2- 172.21.15.195:0/1438009714 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd30002ff60 0x7fd300032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:41.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.477+0000 7fd317fff700 1 --2- 172.21.15.195:0/1438009714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3180f51a0 0x7fd3181064c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:41.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.477+0000 7fd317fff700 1 -- 172.21.15.195:0/1438009714 >> 172.21.15.195:0/1438009714 conn(0x7fd3180f09a0 msgr2=0x7fd3180f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:41.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.477+0000 7fd317fff700 1 -- 172.21.15.195:0/1438009714 shutdown_connections 2023-10-12T22:33:41.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:41.477+0000 7fd317fff700 1 -- 172.21.15.195:0/1438009714 wait complete. 2023-10-12T22:33:41.480 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:42.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:42 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:40.817155+0000 mgr.smithi131.efktma (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:42.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:42 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:41.475401+0000 mon.smithi131 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.195:0/1438009714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:42.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:42.876 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:44.008 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.007+0000 7f216f59c700 1 -- 172.21.15.195:0/1972676419 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21680f6510 msgr2=0x7f21680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:44.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.007+0000 7f216f59c700 1 --2- 172.21.15.195:0/1972676419 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21680f6510 0x7f21680f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f2158009fb0 tx=0x7f2158009b30).stop 2023-10-12T22:33:44.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.007+0000 7f216f59c700 1 -- 172.21.15.195:0/1972676419 shutdown_connections 2023-10-12T22:33:44.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.007+0000 7f216f59c700 1 --2- 172.21.15.195:0/1972676419 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21680f6510 0x7f21680f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:44.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.007+0000 7f216f59c700 1 -- 172.21.15.195:0/1972676419 >> 172.21.15.195:0/1972676419 conn(0x7f21680f1c70 msgr2=0x7f21680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:44.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.007+0000 7f216f59c700 1 -- 172.21.15.195:0/1972676419 shutdown_connections 2023-10-12T22:33:44.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.007+0000 7f216f59c700 1 -- 172.21.15.195:0/1972676419 wait complete. 2023-10-12T22:33:44.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.008+0000 7f216f59c700 1 Processor -- start 2023-10-12T22:33:44.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.008+0000 7f216f59c700 1 -- start start 2023-10-12T22:33:44.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.008+0000 7f216f59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21680f6510 0x7f2168106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:44.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.008+0000 7f216f59c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2168106c20 con 0x7f21680f6510 2023-10-12T22:33:44.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.008+0000 7f216d338700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21680f6510 0x7f2168106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:44.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.008+0000 7f216d338700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21680f6510 0x7f2168106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:50442/0 (socket says 172.21.15.195:50442) 2023-10-12T22:33:44.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.008+0000 7f216d338700 1 -- 172.21.15.195:0/1784472020 learned_addr learned my addr 172.21.15.195:0/1784472020 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:44.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.009+0000 7f216d338700 1 -- 172.21.15.195:0/1784472020 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2158005040 con 0x7f21680f6510 2023-10-12T22:33:44.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.009+0000 7f216d338700 1 --2- 172.21.15.195:0/1784472020 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21680f6510 0x7f2168106730 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f2158013510 tx=0x7f215800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:44.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.009+0000 7f215dffb700 1 -- 172.21.15.195:0/1784472020 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2158009320 con 0x7f21680f6510 2023-10-12T22:33:44.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.009+0000 7f216f59c700 1 -- 172.21.15.195:0/1784472020 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2168106e20 con 0x7f21680f6510 2023-10-12T22:33:44.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.009+0000 7f215dffb700 1 -- 172.21.15.195:0/1784472020 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2158013f00 con 0x7f21680f6510 2023-10-12T22:33:44.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.009+0000 7f215dffb700 1 -- 172.21.15.195:0/1784472020 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2158014d90 con 0x7f21680f6510 2023-10-12T22:33:44.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.009+0000 7f216f59c700 1 -- 172.21.15.195:0/1784472020 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21681072b0 con 0x7f21680f6510 2023-10-12T22:33:44.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.010+0000 7f215dffb700 1 -- 172.21.15.195:0/1784472020 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21580095f0 con 0x7f21680f6510 2023-10-12T22:33:44.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.010+0000 7f216f59c700 1 -- 172.21.15.195:0/1784472020 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2168045ad0 con 0x7f21680f6510 2023-10-12T22:33:44.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.010+0000 7f215dffb700 1 --2- 172.21.15.195:0/1784472020 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f215402ff60 0x7f2154032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:44.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.010+0000 7f215dffb700 1 -- 172.21.15.195:0/1784472020 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2158041000 con 0x7f21680f6510 2023-10-12T22:33:44.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.010+0000 7f216cb37700 1 --2- 172.21.15.195:0/1784472020 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f215402ff60 0x7f2154032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:44.018 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.011+0000 7f216cb37700 1 --2- 172.21.15.195:0/1784472020 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f215402ff60 0x7f2154032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f21640096a0 tx=0x7f2164006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:44.018 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.015+0000 7f215dffb700 1 -- 172.21.15.195:0/1784472020 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2158014420 con 0x7f21680f6510 2023-10-12T22:33:44.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.221+0000 7f216f59c700 1 -- 172.21.15.195:0/1784472020 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21680fbf60 con 0x7f21680f6510 2023-10-12T22:33:44.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.221+0000 7f215dffb700 1 -- 172.21.15.195:0/1784472020 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f215801d7f0 con 0x7f21680f6510 2023-10-12T22:33:44.222 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:44.223 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:44.225 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.223+0000 7f216f59c700 1 -- 172.21.15.195:0/1784472020 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f215402ff60 msgr2=0x7f2154032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:44.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.223+0000 7f216f59c700 1 --2- 172.21.15.195:0/1784472020 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f215402ff60 0x7f2154032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f21640096a0 tx=0x7f2164006b40).stop 2023-10-12T22:33:44.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.223+0000 7f216f59c700 1 -- 172.21.15.195:0/1784472020 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21680f6510 msgr2=0x7f2168106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:44.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.223+0000 7f216f59c700 1 --2- 172.21.15.195:0/1784472020 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21680f6510 0x7f2168106730 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f2158013510 tx=0x7f215800b040).stop 2023-10-12T22:33:44.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.223+0000 7f216f59c700 1 -- 172.21.15.195:0/1784472020 shutdown_connections 2023-10-12T22:33:44.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.223+0000 7f216f59c700 1 --2- 172.21.15.195:0/1784472020 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f215402ff60 0x7f2154032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:44.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.223+0000 7f216f59c700 1 --2- 172.21.15.195:0/1784472020 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21680f6510 0x7f2168106730 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:44.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.223+0000 7f216f59c700 1 -- 172.21.15.195:0/1784472020 >> 172.21.15.195:0/1784472020 conn(0x7f21680f1c70 msgr2=0x7f21680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:44.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.223+0000 7f216f59c700 1 -- 172.21.15.195:0/1784472020 shutdown_connections 2023-10-12T22:33:44.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:44.223+0000 7f216f59c700 1 -- 172.21.15.195:0/1784472020 wait complete. 2023-10-12T22:33:44.227 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:44.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:44 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:42.817527+0000 mgr.smithi131.efktma (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:45.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:45 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:44.221667+0000 mon.smithi131 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.195:0/1784472020' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:45.590 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:45.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:46.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:46 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:44.817911+0000 mgr.smithi131.efktma (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:46.719 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.717+0000 7f6bde588700 1 -- 172.21.15.195:0/319900554 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bd80f51a0 msgr2=0x7f6bd80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:46.719 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.717+0000 7f6bde588700 1 --2- 172.21.15.195:0/319900554 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bd80f51a0 0x7f6bd80f55c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f6bd4009fb0 tx=0x7f6bd4009b30).stop 2023-10-12T22:33:46.719 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.717+0000 7f6bde588700 1 -- 172.21.15.195:0/319900554 shutdown_connections 2023-10-12T22:33:46.720 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.717+0000 7f6bde588700 1 --2- 172.21.15.195:0/319900554 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bd80f51a0 0x7f6bd80f55c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:46.720 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.717+0000 7f6bde588700 1 -- 172.21.15.195:0/319900554 >> 172.21.15.195:0/319900554 conn(0x7f6bd80f09a0 msgr2=0x7f6bd80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:46.720 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.717+0000 7f6bde588700 1 -- 172.21.15.195:0/319900554 shutdown_connections 2023-10-12T22:33:46.720 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.717+0000 7f6bde588700 1 -- 172.21.15.195:0/319900554 wait complete. 2023-10-12T22:33:46.720 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.718+0000 7f6bde588700 1 Processor -- start 2023-10-12T22:33:46.721 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.718+0000 7f6bde588700 1 -- start start 2023-10-12T22:33:46.722 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.718+0000 7f6bde588700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bd80f51a0 0x7f6bd81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:46.723 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.718+0000 7f6bde588700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd8060220 con 0x7f6bd80f51a0 2023-10-12T22:33:46.723 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.718+0000 7f6bdd586700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bd80f51a0 0x7f6bd81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:46.723 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.718+0000 7f6bdd586700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bd80f51a0 0x7f6bd81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:50464/0 (socket says 172.21.15.195:50464) 2023-10-12T22:33:46.723 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.718+0000 7f6bdd586700 1 -- 172.21.15.195:0/3525969930 learned_addr learned my addr 172.21.15.195:0/3525969930 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:46.724 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.719+0000 7f6bdd586700 1 -- 172.21.15.195:0/3525969930 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd4005040 con 0x7f6bd80f51a0 2023-10-12T22:33:46.724 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.719+0000 7f6bdd586700 1 --2- 172.21.15.195:0/3525969930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bd80f51a0 0x7f6bd81091d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6bd4013510 tx=0x7f6bd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:46.724 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.719+0000 7f6bcdffb700 1 -- 172.21.15.195:0/3525969930 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bd4009320 con 0x7f6bd80f51a0 2023-10-12T22:33:46.724 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.719+0000 7f6bde588700 1 -- 172.21.15.195:0/3525969930 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd8060420 con 0x7f6bd80f51a0 2023-10-12T22:33:46.724 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.719+0000 7f6bcdffb700 1 -- 172.21.15.195:0/3525969930 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bd4013f00 con 0x7f6bd80f51a0 2023-10-12T22:33:46.725 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.719+0000 7f6bcdffb700 1 -- 172.21.15.195:0/3525969930 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bd4014dc0 con 0x7f6bd80f51a0 2023-10-12T22:33:46.725 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.719+0000 7f6bde588700 1 -- 172.21.15.195:0/3525969930 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd8109880 con 0x7f6bd80f51a0 2023-10-12T22:33:46.725 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.720+0000 7f6bcdffb700 1 -- 172.21.15.195:0/3525969930 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6bd40095f0 con 0x7f6bd80f51a0 2023-10-12T22:33:46.725 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.720+0000 7f6bde588700 1 -- 172.21.15.195:0/3525969930 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd8059100 con 0x7f6bd80f51a0 2023-10-12T22:33:46.725 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.720+0000 7f6bcdffb700 1 --2- 172.21.15.195:0/3525969930 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f6bc402ff60 0x7f6bc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:46.726 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.720+0000 7f6bcdffb700 1 -- 172.21.15.195:0/3525969930 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6bd4041020 con 0x7f6bd80f51a0 2023-10-12T22:33:46.726 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.720+0000 7f6bdcd85700 1 --2- 172.21.15.195:0/3525969930 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f6bc402ff60 0x7f6bc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:46.726 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.721+0000 7f6bdcd85700 1 --2- 172.21.15.195:0/3525969930 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f6bc402ff60 0x7f6bc4032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6bc80096a0 tx=0x7f6bc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:46.726 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.725+0000 7f6bcdffb700 1 -- 172.21.15.195:0/3525969930 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bd4014420 con 0x7f6bd80f51a0 2023-10-12T22:33:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.933+0000 7f6bde588700 1 -- 172.21.15.195:0/3525969930 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6bd8045ad0 con 0x7f6bd80f51a0 2023-10-12T22:33:46.935 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.934+0000 7f6bcdffb700 1 -- 172.21.15.195:0/3525969930 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6bd401c920 con 0x7f6bd80f51a0 2023-10-12T22:33:46.936 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:46.936 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.935+0000 7f6bde588700 1 -- 172.21.15.195:0/3525969930 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f6bc402ff60 msgr2=0x7f6bc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.936+0000 7f6bde588700 1 --2- 172.21.15.195:0/3525969930 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f6bc402ff60 0x7f6bc4032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6bc80096a0 tx=0x7f6bc8006b40).stop 2023-10-12T22:33:46.937 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.936+0000 7f6bde588700 1 -- 172.21.15.195:0/3525969930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bd80f51a0 msgr2=0x7f6bd81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.936+0000 7f6bde588700 1 --2- 172.21.15.195:0/3525969930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bd80f51a0 0x7f6bd81091d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6bd4013510 tx=0x7f6bd400b040).stop 2023-10-12T22:33:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.936+0000 7f6bde588700 1 -- 172.21.15.195:0/3525969930 shutdown_connections 2023-10-12T22:33:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.936+0000 7f6bde588700 1 --2- 172.21.15.195:0/3525969930 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f6bc402ff60 0x7f6bc4032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.936+0000 7f6bde588700 1 --2- 172.21.15.195:0/3525969930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bd80f51a0 0x7f6bd81091d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.936+0000 7f6bde588700 1 -- 172.21.15.195:0/3525969930 >> 172.21.15.195:0/3525969930 conn(0x7f6bd80f09a0 msgr2=0x7f6bd80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:46.938 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.936+0000 7f6bde588700 1 -- 172.21.15.195:0/3525969930 shutdown_connections 2023-10-12T22:33:46.939 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:46.936+0000 7f6bde588700 1 -- 172.21.15.195:0/3525969930 wait complete. 2023-10-12T22:33:46.939 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:47.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:47 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:46.935448+0000 mon.smithi131 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.195:0/3525969930' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:48.306 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:48.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:48.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:48 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:46.818270+0000 mgr.smithi131.efktma (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:49.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.475+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3281297487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a40f5410 msgr2=0x7fd6a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:49.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.475+0000 7fd6a8b40700 1 --2- 172.21.15.195:0/3281297487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a40f5410 0x7fd6a40f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd694009fb0 tx=0x7fd694009b30).stop 2023-10-12T22:33:49.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.475+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3281297487 shutdown_connections 2023-10-12T22:33:49.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.475+0000 7fd6a8b40700 1 --2- 172.21.15.195:0/3281297487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a40f5410 0x7fd6a40f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:49.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.475+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3281297487 >> 172.21.15.195:0/3281297487 conn(0x7fd6a40f0b70 msgr2=0x7fd6a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:49.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.475+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3281297487 shutdown_connections 2023-10-12T22:33:49.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.475+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3281297487 wait complete. 2023-10-12T22:33:49.479 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.476+0000 7fd6a8b40700 1 Processor -- start 2023-10-12T22:33:49.480 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.476+0000 7fd6a8b40700 1 -- start start 2023-10-12T22:33:49.480 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.476+0000 7fd6a8b40700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a40f5410 0x7fd6a4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:49.480 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.476+0000 7fd6a8b40700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a4060220 con 0x7fd6a40f5410 2023-10-12T22:33:49.480 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.476+0000 7fd6a259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a40f5410 0x7fd6a4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:49.480 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.476+0000 7fd6a259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a40f5410 0x7fd6a4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:44634/0 (socket says 172.21.15.195:44634) 2023-10-12T22:33:49.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.476+0000 7fd6a259c700 1 -- 172.21.15.195:0/3897949080 learned_addr learned my addr 172.21.15.195:0/3897949080 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:49.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.477+0000 7fd6a259c700 1 -- 172.21.15.195:0/3897949080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd694005040 con 0x7fd6a40f5410 2023-10-12T22:33:49.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.477+0000 7fd6a259c700 1 --2- 172.21.15.195:0/3897949080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a40f5410 0x7fd6a4109450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd694014000 tx=0x7fd69400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:49.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.477+0000 7fd69affd700 1 -- 172.21.15.195:0/3897949080 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6940095f0 con 0x7fd6a40f5410 2023-10-12T22:33:49.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.477+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3897949080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a4060420 con 0x7fd6a40f5410 2023-10-12T22:33:49.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.477+0000 7fd69affd700 1 -- 172.21.15.195:0/3897949080 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd694007e70 con 0x7fd6a40f5410 2023-10-12T22:33:49.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.477+0000 7fd69affd700 1 -- 172.21.15.195:0/3897949080 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd694014ea0 con 0x7fd6a40f5410 2023-10-12T22:33:49.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.477+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3897949080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a4109b00 con 0x7fd6a40f5410 2023-10-12T22:33:49.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.478+0000 7fd69affd700 1 -- 172.21.15.195:0/3897949080 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd694025070 con 0x7fd6a40f5410 2023-10-12T22:33:49.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.478+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3897949080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a4059100 con 0x7fd6a40f5410 2023-10-12T22:33:49.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.478+0000 7fd69affd700 1 --2- 172.21.15.195:0/3897949080 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd69002ff60 0x7fd690032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:49.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.478+0000 7fd69affd700 1 -- 172.21.15.195:0/3897949080 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd694040940 con 0x7fd6a40f5410 2023-10-12T22:33:49.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.478+0000 7fd6a1d9b700 1 --2- 172.21.15.195:0/3897949080 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd69002ff60 0x7fd690032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:49.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.479+0000 7fd6a1d9b700 1 --2- 172.21.15.195:0/3897949080 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd69002ff60 0x7fd690032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fd68c0096a0 tx=0x7fd68c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:49.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.483+0000 7fd69affd700 1 -- 172.21.15.195:0/3897949080 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd6940079c0 con 0x7fd6a40f5410 2023-10-12T22:33:49.688 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.686+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3897949080 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6a4045ad0 con 0x7fd6a40f5410 2023-10-12T22:33:49.688 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.687+0000 7fd69affd700 1 -- 172.21.15.195:0/3897949080 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd69400f300 con 0x7fd6a40f5410 2023-10-12T22:33:49.689 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:49.689 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:49.690 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.689+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3897949080 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd69002ff60 msgr2=0x7fd690032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:49.690 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.689+0000 7fd6a8b40700 1 --2- 172.21.15.195:0/3897949080 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd69002ff60 0x7fd690032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fd68c0096a0 tx=0x7fd68c006b40).stop 2023-10-12T22:33:49.690 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.689+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3897949080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a40f5410 msgr2=0x7fd6a4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:49.691 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.689+0000 7fd6a8b40700 1 --2- 172.21.15.195:0/3897949080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a40f5410 0x7fd6a4109450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd694014000 tx=0x7fd69400b040).stop 2023-10-12T22:33:49.691 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.689+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3897949080 shutdown_connections 2023-10-12T22:33:49.691 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.689+0000 7fd6a8b40700 1 --2- 172.21.15.195:0/3897949080 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd69002ff60 0x7fd690032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:49.691 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.689+0000 7fd6a8b40700 1 --2- 172.21.15.195:0/3897949080 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a40f5410 0x7fd6a4109450 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:49.691 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.689+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3897949080 >> 172.21.15.195:0/3897949080 conn(0x7fd6a40f0b70 msgr2=0x7fd6a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:49.692 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.689+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3897949080 shutdown_connections 2023-10-12T22:33:49.692 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:49.689+0000 7fd6a8b40700 1 -- 172.21.15.195:0/3897949080 wait complete. 2023-10-12T22:33:49.692 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:50.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:48.818563+0000 mgr.smithi131.efktma (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:50.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:49.688536+0000 mon.smithi131 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.195:0/3897949080' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:51.096 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:51.097 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:52.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.263+0000 7f1356c55700 1 -- 172.21.15.195:0/2686470367 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13500f62a0 msgr2=0x7f13500f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:52.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.263+0000 7f1356c55700 1 --2- 172.21.15.195:0/2686470367 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13500f62a0 0x7f13500f66c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f1340009fb0 tx=0x7f1340009b30).stop 2023-10-12T22:33:52.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.263+0000 7f1356c55700 1 -- 172.21.15.195:0/2686470367 shutdown_connections 2023-10-12T22:33:52.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.263+0000 7f1356c55700 1 --2- 172.21.15.195:0/2686470367 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13500f62a0 0x7f13500f66c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:52.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.263+0000 7f1356c55700 1 -- 172.21.15.195:0/2686470367 >> 172.21.15.195:0/2686470367 conn(0x7f13500f1aa0 msgr2=0x7f13500f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:52.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.263+0000 7f1356c55700 1 -- 172.21.15.195:0/2686470367 shutdown_connections 2023-10-12T22:33:52.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.264+0000 7f1356c55700 1 -- 172.21.15.195:0/2686470367 wait complete. 2023-10-12T22:33:52.268 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.264+0000 7f1356c55700 1 Processor -- start 2023-10-12T22:33:52.268 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.264+0000 7f1356c55700 1 -- start start 2023-10-12T22:33:52.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.264+0000 7f1356c55700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13500f62a0 0x7f13501064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:52.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.264+0000 7f1356c55700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13501069a0 con 0x7f13500f62a0 2023-10-12T22:33:52.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.265+0000 7f1355c53700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13500f62a0 0x7f13501064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:52.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.265+0000 7f1355c53700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13500f62a0 0x7f13501064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:44648/0 (socket says 172.21.15.195:44648) 2023-10-12T22:33:52.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.265+0000 7f1355c53700 1 -- 172.21.15.195:0/4155825183 learned_addr learned my addr 172.21.15.195:0/4155825183 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:52.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.265+0000 7f1355c53700 1 -- 172.21.15.195:0/4155825183 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1340005040 con 0x7f13500f62a0 2023-10-12T22:33:52.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.265+0000 7f1355c53700 1 --2- 172.21.15.195:0/4155825183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13500f62a0 0x7f13501064b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f1340013510 tx=0x7f134000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:52.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.265+0000 7f13467fc700 1 -- 172.21.15.195:0/4155825183 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1340009320 con 0x7f13500f62a0 2023-10-12T22:33:52.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.265+0000 7f1356c55700 1 -- 172.21.15.195:0/4155825183 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1350106ba0 con 0x7f13500f62a0 2023-10-12T22:33:52.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.265+0000 7f13467fc700 1 -- 172.21.15.195:0/4155825183 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1340013f00 con 0x7f13500f62a0 2023-10-12T22:33:52.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.265+0000 7f13467fc700 1 -- 172.21.15.195:0/4155825183 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1340014dc0 con 0x7f13500f62a0 2023-10-12T22:33:52.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.265+0000 7f1356c55700 1 -- 172.21.15.195:0/4155825183 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1350107030 con 0x7f13500f62a0 2023-10-12T22:33:52.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.266+0000 7f13467fc700 1 -- 172.21.15.195:0/4155825183 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13400095f0 con 0x7f13500f62a0 2023-10-12T22:33:52.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.266+0000 7f1356c55700 1 -- 172.21.15.195:0/4155825183 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1350045ad0 con 0x7f13500f62a0 2023-10-12T22:33:52.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.266+0000 7f13467fc700 1 --2- 172.21.15.195:0/4155825183 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f133c02ff60 0x7f133c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:52.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.266+0000 7f13467fc700 1 -- 172.21.15.195:0/4155825183 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1340041000 con 0x7f13500f62a0 2023-10-12T22:33:52.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.266+0000 7f1355452700 1 --2- 172.21.15.195:0/4155825183 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f133c02ff60 0x7f133c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:52.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.267+0000 7f1355452700 1 --2- 172.21.15.195:0/4155825183 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f133c02ff60 0x7f133c032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f134c0096a0 tx=0x7f134c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:52.273 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.271+0000 7f13467fc700 1 -- 172.21.15.195:0/4155825183 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f134001c790 con 0x7f13500f62a0 2023-10-12T22:33:52.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.476+0000 7f1356c55700 1 -- 172.21.15.195:0/4155825183 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1350107ee0 con 0x7f13500f62a0 2023-10-12T22:33:52.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.477+0000 7f13467fc700 1 -- 172.21.15.195:0/4155825183 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1340010230 con 0x7f13500f62a0 2023-10-12T22:33:52.478 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:52.479 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:52.480 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.479+0000 7f1356c55700 1 -- 172.21.15.195:0/4155825183 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f133c02ff60 msgr2=0x7f133c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:52.480 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.479+0000 7f1356c55700 1 --2- 172.21.15.195:0/4155825183 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f133c02ff60 0x7f133c032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f134c0096a0 tx=0x7f134c006b40).stop 2023-10-12T22:33:52.480 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.479+0000 7f1356c55700 1 -- 172.21.15.195:0/4155825183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13500f62a0 msgr2=0x7f13501064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:52.480 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.479+0000 7f1356c55700 1 --2- 172.21.15.195:0/4155825183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13500f62a0 0x7f13501064b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f1340013510 tx=0x7f134000b040).stop 2023-10-12T22:33:52.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.479+0000 7f1356c55700 1 -- 172.21.15.195:0/4155825183 shutdown_connections 2023-10-12T22:33:52.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.479+0000 7f1356c55700 1 --2- 172.21.15.195:0/4155825183 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f133c02ff60 0x7f133c032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:52.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.479+0000 7f1356c55700 1 --2- 172.21.15.195:0/4155825183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13500f62a0 0x7f13501064b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:52.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.479+0000 7f1356c55700 1 -- 172.21.15.195:0/4155825183 >> 172.21.15.195:0/4155825183 conn(0x7f13500f1aa0 msgr2=0x7f13500f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:52.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.479+0000 7f1356c55700 1 -- 172.21.15.195:0/4155825183 shutdown_connections 2023-10-12T22:33:52.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:52.479+0000 7f1356c55700 1 -- 172.21.15.195:0/4155825183 wait complete. 2023-10-12T22:33:52.482 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:52.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:50.818919+0000 mgr.smithi131.efktma (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:53.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:53 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:52.478387+0000 mon.smithi131 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.195:0/4155825183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:53.856 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:53.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:54.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:52.819304+0000 mgr.smithi131.efktma (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:55.000 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.995+0000 7fba30516700 1 -- 172.21.15.195:0/913744391 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba280f5410 msgr2=0x7fba280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:55.001 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.995+0000 7fba30516700 1 --2- 172.21.15.195:0/913744391 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba280f5410 0x7fba280f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fba1c009fb0 tx=0x7fba1c009b30).stop 2023-10-12T22:33:55.001 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.995+0000 7fba30516700 1 -- 172.21.15.195:0/913744391 shutdown_connections 2023-10-12T22:33:55.001 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.995+0000 7fba30516700 1 --2- 172.21.15.195:0/913744391 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba280f5410 0x7fba280f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:55.002 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.995+0000 7fba30516700 1 -- 172.21.15.195:0/913744391 >> 172.21.15.195:0/913744391 conn(0x7fba280f0b70 msgr2=0x7fba280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:55.002 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.995+0000 7fba30516700 1 -- 172.21.15.195:0/913744391 shutdown_connections 2023-10-12T22:33:55.002 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.996+0000 7fba30516700 1 -- 172.21.15.195:0/913744391 wait complete. 2023-10-12T22:33:55.003 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.996+0000 7fba30516700 1 Processor -- start 2023-10-12T22:33:55.003 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.996+0000 7fba30516700 1 -- start start 2023-10-12T22:33:55.003 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.996+0000 7fba30516700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba280f5410 0x7fba28106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:55.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.996+0000 7fba30516700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba28106c10 con 0x7fba280f5410 2023-10-12T22:33:55.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.996+0000 7fba2e2b2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba280f5410 0x7fba28106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:55.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.996+0000 7fba2e2b2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba280f5410 0x7fba28106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:44672/0 (socket says 172.21.15.195:44672) 2023-10-12T22:33:55.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.996+0000 7fba2e2b2700 1 -- 172.21.15.195:0/975491827 learned_addr learned my addr 172.21.15.195:0/975491827 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:55.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.997+0000 7fba2e2b2700 1 -- 172.21.15.195:0/975491827 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c005040 con 0x7fba280f5410 2023-10-12T22:33:55.005 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.997+0000 7fba2e2b2700 1 --2- 172.21.15.195:0/975491827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba280f5410 0x7fba28106720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fba1c013510 tx=0x7fba1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:55.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.997+0000 7fba1affd700 1 -- 172.21.15.195:0/975491827 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba1c009320 con 0x7fba280f5410 2023-10-12T22:33:55.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.997+0000 7fba1affd700 1 -- 172.21.15.195:0/975491827 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba1c013f00 con 0x7fba280f5410 2023-10-12T22:33:55.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.997+0000 7fba30516700 1 -- 172.21.15.195:0/975491827 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba28106e10 con 0x7fba280f5410 2023-10-12T22:33:55.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.997+0000 7fba1affd700 1 -- 172.21.15.195:0/975491827 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba1c014d90 con 0x7fba280f5410 2023-10-12T22:33:55.007 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.997+0000 7fba30516700 1 -- 172.21.15.195:0/975491827 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba281072a0 con 0x7fba280f5410 2023-10-12T22:33:55.007 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.998+0000 7fba1affd700 1 -- 172.21.15.195:0/975491827 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba1c014420 con 0x7fba280f5410 2023-10-12T22:33:55.007 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.998+0000 7fba30516700 1 -- 172.21.15.195:0/975491827 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba28045ad0 con 0x7fba280f5410 2023-10-12T22:33:55.008 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.998+0000 7fba1affd700 1 --2- 172.21.15.195:0/975491827 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fba1402ff60 0x7fba14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:55.008 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.998+0000 7fba1affd700 1 -- 172.21.15.195:0/975491827 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba1c041110 con 0x7fba280f5410 2023-10-12T22:33:55.008 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.998+0000 7fba2dab1700 1 --2- 172.21.15.195:0/975491827 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fba1402ff60 0x7fba14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:55.008 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:54.999+0000 7fba2dab1700 1 --2- 172.21.15.195:0/975491827 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fba1402ff60 0x7fba14032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fba240096a0 tx=0x7fba24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:55.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.003+0000 7fba1affd700 1 -- 172.21.15.195:0/975491827 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba1c0095f0 con 0x7fba280f5410 2023-10-12T22:33:55.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.212+0000 7fba30516700 1 -- 172.21.15.195:0/975491827 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba281080f0 con 0x7fba280f5410 2023-10-12T22:33:55.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.213+0000 7fba1affd700 1 -- 172.21.15.195:0/975491827 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fba1c01db70 con 0x7fba280f5410 2023-10-12T22:33:55.215 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:55.215 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:55.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.215+0000 7fba30516700 1 -- 172.21.15.195:0/975491827 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fba1402ff60 msgr2=0x7fba14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:55.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.215+0000 7fba30516700 1 --2- 172.21.15.195:0/975491827 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fba1402ff60 0x7fba14032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fba240096a0 tx=0x7fba24006b40).stop 2023-10-12T22:33:55.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.215+0000 7fba30516700 1 -- 172.21.15.195:0/975491827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba280f5410 msgr2=0x7fba28106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:55.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.215+0000 7fba30516700 1 --2- 172.21.15.195:0/975491827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba280f5410 0x7fba28106720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fba1c013510 tx=0x7fba1c00b040).stop 2023-10-12T22:33:55.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.215+0000 7fba30516700 1 -- 172.21.15.195:0/975491827 shutdown_connections 2023-10-12T22:33:55.218 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.215+0000 7fba30516700 1 --2- 172.21.15.195:0/975491827 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fba1402ff60 0x7fba14032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:55.218 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.215+0000 7fba30516700 1 --2- 172.21.15.195:0/975491827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba280f5410 0x7fba28106720 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:55.218 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.215+0000 7fba30516700 1 -- 172.21.15.195:0/975491827 >> 172.21.15.195:0/975491827 conn(0x7fba280f0b70 msgr2=0x7fba2805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:55.218 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.215+0000 7fba30516700 1 -- 172.21.15.195:0/975491827 shutdown_connections 2023-10-12T22:33:55.219 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:55.215+0000 7fba30516700 1 -- 172.21.15.195:0/975491827 wait complete. 2023-10-12T22:33:55.219 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:56.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:54.819628+0000 mgr.smithi131.efktma (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:56.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:55.214650+0000 mon.smithi131 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.195:0/975491827' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:56.590 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:56.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:33:57.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.707+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3929745670 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc2e40601e0 msgr2=0x7fc2e40f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:57.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.707+0000 7fc2e9ece700 1 --2- 172.21.15.195:0/3929745670 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc2e40601e0 0x7fc2e40f8f20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fc2d8009fb0 tx=0x7fc2d8009b30).stop 2023-10-12T22:33:57.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.707+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3929745670 shutdown_connections 2023-10-12T22:33:57.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.707+0000 7fc2e9ece700 1 --2- 172.21.15.195:0/3929745670 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc2e40601e0 0x7fc2e40f8f20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:57.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.707+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3929745670 >> 172.21.15.195:0/3929745670 conn(0x7fc2e40f09a0 msgr2=0x7fc2e40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:57.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.707+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3929745670 shutdown_connections 2023-10-12T22:33:57.710 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.707+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3929745670 wait complete. 2023-10-12T22:33:57.710 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.707+0000 7fc2e9ece700 1 Processor -- start 2023-10-12T22:33:57.710 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.707+0000 7fc2e9ece700 1 -- start start 2023-10-12T22:33:57.712 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.708+0000 7fc2e9ece700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc2e40601e0 0x7fc2e4106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:57.712 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.708+0000 7fc2e9ece700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e4106a10 con 0x7fc2e40601e0 2023-10-12T22:33:57.712 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.708+0000 7fc2e8ecc700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc2e40601e0 0x7fc2e4106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:57.713 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.708+0000 7fc2e8ecc700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc2e40601e0 0x7fc2e4106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:44690/0 (socket says 172.21.15.195:44690) 2023-10-12T22:33:57.713 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.708+0000 7fc2e8ecc700 1 -- 172.21.15.195:0/3893877860 learned_addr learned my addr 172.21.15.195:0/3893877860 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:33:57.713 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.708+0000 7fc2e8ecc700 1 -- 172.21.15.195:0/3893877860 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d8005040 con 0x7fc2e40601e0 2023-10-12T22:33:57.713 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.708+0000 7fc2e8ecc700 1 --2- 172.21.15.195:0/3893877860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc2e40601e0 0x7fc2e4106520 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fc2d8013510 tx=0x7fc2d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:57.713 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.709+0000 7fc2e17fa700 1 -- 172.21.15.195:0/3893877860 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2d8009320 con 0x7fc2e40601e0 2023-10-12T22:33:57.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.709+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3893877860 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e4106c10 con 0x7fc2e40601e0 2023-10-12T22:33:57.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.709+0000 7fc2e17fa700 1 -- 172.21.15.195:0/3893877860 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2d8013f00 con 0x7fc2e40601e0 2023-10-12T22:33:57.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.709+0000 7fc2e17fa700 1 -- 172.21.15.195:0/3893877860 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2d8014d90 con 0x7fc2e40601e0 2023-10-12T22:33:57.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.709+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3893877860 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e41070a0 con 0x7fc2e40601e0 2023-10-12T22:33:57.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.709+0000 7fc2e17fa700 1 -- 172.21.15.195:0/3893877860 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2d8014420 con 0x7fc2e40601e0 2023-10-12T22:33:57.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.709+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3893877860 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e4045ad0 con 0x7fc2e40601e0 2023-10-12T22:33:57.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.710+0000 7fc2e17fa700 1 --2- 172.21.15.195:0/3893877860 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc2d402ff60 0x7fc2d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:33:57.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.710+0000 7fc2e17fa700 1 -- 172.21.15.195:0/3893877860 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2d8041000 con 0x7fc2e40601e0 2023-10-12T22:33:57.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.710+0000 7fc2e3fff700 1 --2- 172.21.15.195:0/3893877860 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc2d402ff60 0x7fc2d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:33:57.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.710+0000 7fc2e3fff700 1 --2- 172.21.15.195:0/3893877860 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc2d402ff60 0x7fc2d4032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fc2d00096a0 tx=0x7fc2d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:33:57.716 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.714+0000 7fc2e17fa700 1 -- 172.21.15.195:0/3893877860 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2d80095f0 con 0x7fc2e40601e0 2023-10-12T22:33:57.923 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.922+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3893877860 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2e4107f50 con 0x7fc2e40601e0 2023-10-12T22:33:57.923 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.922+0000 7fc2e17fa700 1 -- 172.21.15.195:0/3893877860 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc2d801db70 con 0x7fc2e40601e0 2023-10-12T22:33:57.924 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:33:57.924 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:33:57.925 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.924+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3893877860 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc2d402ff60 msgr2=0x7fc2d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:57.925 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.924+0000 7fc2e9ece700 1 --2- 172.21.15.195:0/3893877860 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc2d402ff60 0x7fc2d4032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fc2d00096a0 tx=0x7fc2d0006b40).stop 2023-10-12T22:33:57.925 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.924+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3893877860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc2e40601e0 msgr2=0x7fc2e4106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:33:57.926 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.924+0000 7fc2e9ece700 1 --2- 172.21.15.195:0/3893877860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc2e40601e0 0x7fc2e4106520 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fc2d8013510 tx=0x7fc2d800b040).stop 2023-10-12T22:33:57.926 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.924+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3893877860 shutdown_connections 2023-10-12T22:33:57.926 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.924+0000 7fc2e9ece700 1 --2- 172.21.15.195:0/3893877860 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc2d402ff60 0x7fc2d4032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:57.926 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.924+0000 7fc2e9ece700 1 --2- 172.21.15.195:0/3893877860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc2e40601e0 0x7fc2e4106520 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:33:57.926 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.924+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3893877860 >> 172.21.15.195:0/3893877860 conn(0x7fc2e40f09a0 msgr2=0x7fc2e40f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:33:57.927 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.924+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3893877860 shutdown_connections 2023-10-12T22:33:57.927 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:33:57.924+0000 7fc2e9ece700 1 -- 172.21.15.195:0/3893877860 wait complete. 2023-10-12T22:33:57.927 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:33:58.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:58 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:56.819976+0000 mgr.smithi131.efktma (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:33:58.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:33:58 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:33:57.923750+0000 mon.smithi131 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.195:0/3893877860' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:33:59.299 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:33:59.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:00.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.478+0000 7f8843e59700 1 -- 172.21.15.195:0/962425934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f883c0f6510 msgr2=0x7f883c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:00.481 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.478+0000 7f8843e59700 1 --2- 172.21.15.195:0/962425934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f883c0f6510 0x7f883c0f6930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f882c009fb0 tx=0x7f882c009b30).stop 2023-10-12T22:34:00.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.478+0000 7f8843e59700 1 -- 172.21.15.195:0/962425934 shutdown_connections 2023-10-12T22:34:00.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.478+0000 7f8843e59700 1 --2- 172.21.15.195:0/962425934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f883c0f6510 0x7f883c0f6930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:00.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.478+0000 7f8843e59700 1 -- 172.21.15.195:0/962425934 >> 172.21.15.195:0/962425934 conn(0x7f883c0f1c70 msgr2=0x7f883c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:00.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.478+0000 7f8843e59700 1 -- 172.21.15.195:0/962425934 shutdown_connections 2023-10-12T22:34:00.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.478+0000 7f8843e59700 1 -- 172.21.15.195:0/962425934 wait complete. 2023-10-12T22:34:00.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.479+0000 7f8843e59700 1 Processor -- start 2023-10-12T22:34:00.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.479+0000 7f8843e59700 1 -- start start 2023-10-12T22:34:00.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.479+0000 7f8843e59700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f883c106840 0x7f883c106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:00.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.479+0000 7f8843e59700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c107150 con 0x7f883c106840 2023-10-12T22:34:00.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.479+0000 7f8841bf5700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f883c106840 0x7f883c106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:00.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.479+0000 7f8841bf5700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f883c106840 0x7f883c106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:53644/0 (socket says 172.21.15.195:53644) 2023-10-12T22:34:00.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.479+0000 7f8841bf5700 1 -- 172.21.15.195:0/2943021712 learned_addr learned my addr 172.21.15.195:0/2943021712 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:00.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.480+0000 7f8841bf5700 1 -- 172.21.15.195:0/2943021712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c005040 con 0x7f883c106840 2023-10-12T22:34:00.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.480+0000 7f8841bf5700 1 --2- 172.21.15.195:0/2943021712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f883c106840 0x7f883c106c60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f882c009f80 tx=0x7f882c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:00.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.480+0000 7f88327fc700 1 -- 172.21.15.195:0/2943021712 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f882c0136c0 con 0x7f883c106840 2023-10-12T22:34:00.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.480+0000 7f8843e59700 1 -- 172.21.15.195:0/2943021712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883c107350 con 0x7f883c106840 2023-10-12T22:34:00.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.480+0000 7f88327fc700 1 -- 172.21.15.195:0/2943021712 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f882c013e70 con 0x7f883c106840 2023-10-12T22:34:00.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.480+0000 7f88327fc700 1 -- 172.21.15.195:0/2943021712 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f882c014420 con 0x7f883c106840 2023-10-12T22:34:00.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.480+0000 7f8843e59700 1 -- 172.21.15.195:0/2943021712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883c107ea0 con 0x7f883c106840 2023-10-12T22:34:00.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.481+0000 7f88327fc700 1 -- 172.21.15.195:0/2943021712 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f882c025060 con 0x7f883c106840 2023-10-12T22:34:00.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.481+0000 7f8843e59700 1 -- 172.21.15.195:0/2943021712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883c059100 con 0x7f883c106840 2023-10-12T22:34:00.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.481+0000 7f88327fc700 1 --2- 172.21.15.195:0/2943021712 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f882802ff60 0x7f8828032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:00.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.481+0000 7f88327fc700 1 -- 172.21.15.195:0/2943021712 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f882c040ec0 con 0x7f883c106840 2023-10-12T22:34:00.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.481+0000 7f88413f4700 1 --2- 172.21.15.195:0/2943021712 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f882802ff60 0x7f8828032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:00.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.481+0000 7f88413f4700 1 --2- 172.21.15.195:0/2943021712 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f882802ff60 0x7f8828032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f88380096a0 tx=0x7f8838006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:00.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.486+0000 7f88327fc700 1 -- 172.21.15.195:0/2943021712 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f882c01cd10 con 0x7f883c106840 2023-10-12T22:34:00.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:33:58.820378+0000 mgr.smithi131.efktma (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:00.692 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.690+0000 7f8843e59700 1 -- 172.21.15.195:0/2943021712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f883c045ad0 con 0x7f883c106840 2023-10-12T22:34:00.692 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.691+0000 7f88327fc700 1 -- 172.21.15.195:0/2943021712 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f882c02d2d0 con 0x7f883c106840 2023-10-12T22:34:00.693 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:00.693 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:00.694 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.693+0000 7f8843e59700 1 -- 172.21.15.195:0/2943021712 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f882802ff60 msgr2=0x7f8828032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:00.694 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.693+0000 7f8843e59700 1 --2- 172.21.15.195:0/2943021712 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f882802ff60 0x7f8828032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f88380096a0 tx=0x7f8838006b40).stop 2023-10-12T22:34:00.694 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.693+0000 7f8843e59700 1 -- 172.21.15.195:0/2943021712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f883c106840 msgr2=0x7f883c106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:00.695 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.693+0000 7f8843e59700 1 --2- 172.21.15.195:0/2943021712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f883c106840 0x7f883c106c60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f882c009f80 tx=0x7f882c009b30).stop 2023-10-12T22:34:00.695 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.693+0000 7f8843e59700 1 -- 172.21.15.195:0/2943021712 shutdown_connections 2023-10-12T22:34:00.695 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.693+0000 7f8843e59700 1 --2- 172.21.15.195:0/2943021712 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f882802ff60 0x7f8828032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:00.695 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.693+0000 7f8843e59700 1 --2- 172.21.15.195:0/2943021712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f883c106840 0x7f883c106c60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:00.695 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.693+0000 7f8843e59700 1 -- 172.21.15.195:0/2943021712 >> 172.21.15.195:0/2943021712 conn(0x7f883c0f1c70 msgr2=0x7f883c0f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:00.695 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.693+0000 7f8843e59700 1 -- 172.21.15.195:0/2943021712 shutdown_connections 2023-10-12T22:34:00.696 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:00.693+0000 7f8843e59700 1 -- 172.21.15.195:0/2943021712 wait complete. 2023-10-12T22:34:00.696 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:01.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:01 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:00.692523+0000 mon.smithi131 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.195:0/2943021712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:02.076 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:02.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:02.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:02 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:00.820724+0000 mgr.smithi131.efktma (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:03.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.211+0000 7ff323679700 1 -- 172.21.15.195:0/2919523661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff31c0f6510 msgr2=0x7ff31c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:03.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.211+0000 7ff323679700 1 --2- 172.21.15.195:0/2919523661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff31c0f6510 0x7ff31c0f6930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7ff30c009fb0 tx=0x7ff30c009b30).stop 2023-10-12T22:34:03.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.211+0000 7ff323679700 1 -- 172.21.15.195:0/2919523661 shutdown_connections 2023-10-12T22:34:03.215 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.211+0000 7ff323679700 1 --2- 172.21.15.195:0/2919523661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff31c0f6510 0x7ff31c0f6930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:03.215 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.211+0000 7ff323679700 1 -- 172.21.15.195:0/2919523661 >> 172.21.15.195:0/2919523661 conn(0x7ff31c0f1c70 msgr2=0x7ff31c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:03.215 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.211+0000 7ff323679700 1 -- 172.21.15.195:0/2919523661 shutdown_connections 2023-10-12T22:34:03.215 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.211+0000 7ff323679700 1 -- 172.21.15.195:0/2919523661 wait complete. 2023-10-12T22:34:03.215 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.212+0000 7ff323679700 1 Processor -- start 2023-10-12T22:34:03.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.212+0000 7ff323679700 1 -- start start 2023-10-12T22:34:03.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.212+0000 7ff323679700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff31c0f6510 0x7ff31c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:03.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.212+0000 7ff323679700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c069e20 con 0x7ff31c0f6510 2023-10-12T22:34:03.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.212+0000 7ff321415700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff31c0f6510 0x7ff31c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:03.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.212+0000 7ff321415700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff31c0f6510 0x7ff31c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:53656/0 (socket says 172.21.15.195:53656) 2023-10-12T22:34:03.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.212+0000 7ff321415700 1 -- 172.21.15.195:0/508289431 learned_addr learned my addr 172.21.15.195:0/508289431 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:03.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.212+0000 7ff321415700 1 -- 172.21.15.195:0/508289431 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c005040 con 0x7ff31c0f6510 2023-10-12T22:34:03.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.212+0000 7ff321415700 1 --2- 172.21.15.195:0/508289431 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff31c0f6510 0x7ff31c06b9a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ff30c013510 tx=0x7ff30c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:03.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.213+0000 7ff311ffb700 1 -- 172.21.15.195:0/508289431 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff30c009320 con 0x7ff31c0f6510 2023-10-12T22:34:03.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.213+0000 7ff311ffb700 1 -- 172.21.15.195:0/508289431 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff30c013f00 con 0x7ff31c0f6510 2023-10-12T22:34:03.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.213+0000 7ff323679700 1 -- 172.21.15.195:0/508289431 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31c069fc0 con 0x7ff31c0f6510 2023-10-12T22:34:03.218 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.213+0000 7ff311ffb700 1 -- 172.21.15.195:0/508289431 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff30c014d90 con 0x7ff31c0f6510 2023-10-12T22:34:03.218 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.213+0000 7ff323679700 1 -- 172.21.15.195:0/508289431 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31c06a450 con 0x7ff31c0f6510 2023-10-12T22:34:03.219 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.214+0000 7ff311ffb700 1 -- 172.21.15.195:0/508289431 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff30c0095f0 con 0x7ff31c0f6510 2023-10-12T22:34:03.219 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.214+0000 7ff323679700 1 -- 172.21.15.195:0/508289431 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31c059100 con 0x7ff31c0f6510 2023-10-12T22:34:03.219 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.214+0000 7ff311ffb700 1 --2- 172.21.15.195:0/508289431 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff30802ffb0 0x7ff308032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:03.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.214+0000 7ff311ffb700 1 -- 172.21.15.195:0/508289431 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff30c040fe0 con 0x7ff31c0f6510 2023-10-12T22:34:03.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.214+0000 7ff320c14700 1 --2- 172.21.15.195:0/508289431 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff30802ffb0 0x7ff308032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:03.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.215+0000 7ff320c14700 1 --2- 172.21.15.195:0/508289431 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff30802ffb0 0x7ff308032470 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff3180096a0 tx=0x7ff318006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:03.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.219+0000 7ff311ffb700 1 -- 172.21.15.195:0/508289431 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff30c014420 con 0x7ff31c0f6510 2023-10-12T22:34:03.425 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.423+0000 7ff323679700 1 -- 172.21.15.195:0/508289431 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff31c0fbf60 con 0x7ff31c0f6510 2023-10-12T22:34:03.425 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.424+0000 7ff311ffb700 1 -- 172.21.15.195:0/508289431 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff30c01d7f0 con 0x7ff31c0f6510 2023-10-12T22:34:03.426 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:03.426 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:03.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.426+0000 7ff323679700 1 -- 172.21.15.195:0/508289431 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff30802ffb0 msgr2=0x7ff308032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:03.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.426+0000 7ff323679700 1 --2- 172.21.15.195:0/508289431 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff30802ffb0 0x7ff308032470 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff3180096a0 tx=0x7ff318006b40).stop 2023-10-12T22:34:03.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.426+0000 7ff323679700 1 -- 172.21.15.195:0/508289431 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff31c0f6510 msgr2=0x7ff31c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:03.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.426+0000 7ff323679700 1 --2- 172.21.15.195:0/508289431 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff31c0f6510 0x7ff31c06b9a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ff30c013510 tx=0x7ff30c00b040).stop 2023-10-12T22:34:03.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.426+0000 7ff323679700 1 -- 172.21.15.195:0/508289431 shutdown_connections 2023-10-12T22:34:03.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.426+0000 7ff323679700 1 --2- 172.21.15.195:0/508289431 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff30802ffb0 0x7ff308032470 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:03.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.426+0000 7ff323679700 1 --2- 172.21.15.195:0/508289431 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff31c0f6510 0x7ff31c06b9a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:03.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.426+0000 7ff323679700 1 -- 172.21.15.195:0/508289431 >> 172.21.15.195:0/508289431 conn(0x7ff31c0f1c70 msgr2=0x7ff31c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.426+0000 7ff323679700 1 -- 172.21.15.195:0/508289431 shutdown_connections 2023-10-12T22:34:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:03.426+0000 7ff323679700 1 -- 172.21.15.195:0/508289431 wait complete. 2023-10-12T22:34:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:04.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:02.821053+0000 mgr.smithi131.efktma (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:04.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:03.425467+0000 mon.smithi131 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.195:0/508289431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:04.810 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:04.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:05.983 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.980+0000 7f0f0f722700 1 -- 172.21.15.195:0/348806581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f080f6510 msgr2=0x7f0f080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:05.983 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.980+0000 7f0f0f722700 1 --2- 172.21.15.195:0/348806581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f080f6510 0x7f0f080f6930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0f04009fb0 tx=0x7f0f04009b30).stop 2023-10-12T22:34:05.983 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.980+0000 7f0f0f722700 1 -- 172.21.15.195:0/348806581 shutdown_connections 2023-10-12T22:34:05.983 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.980+0000 7f0f0f722700 1 --2- 172.21.15.195:0/348806581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f080f6510 0x7f0f080f6930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:05.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.980+0000 7f0f0f722700 1 -- 172.21.15.195:0/348806581 >> 172.21.15.195:0/348806581 conn(0x7f0f080f1c70 msgr2=0x7f0f080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:05.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.980+0000 7f0f0f722700 1 -- 172.21.15.195:0/348806581 shutdown_connections 2023-10-12T22:34:05.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.980+0000 7f0f0f722700 1 -- 172.21.15.195:0/348806581 wait complete. 2023-10-12T22:34:05.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.981+0000 7f0f0f722700 1 Processor -- start 2023-10-12T22:34:05.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.981+0000 7f0f0f722700 1 -- start start 2023-10-12T22:34:05.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.981+0000 7f0f0f722700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f080f6510 0x7f0f08106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:05.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.981+0000 7f0f0f722700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f08106c20 con 0x7f0f080f6510 2023-10-12T22:34:05.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.981+0000 7f0f0d4be700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f080f6510 0x7f0f08106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:05.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.981+0000 7f0f0d4be700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f080f6510 0x7f0f08106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:53686/0 (socket says 172.21.15.195:53686) 2023-10-12T22:34:05.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.981+0000 7f0f0d4be700 1 -- 172.21.15.195:0/3096195648 learned_addr learned my addr 172.21.15.195:0/3096195648 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:05.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.981+0000 7f0f0d4be700 1 -- 172.21.15.195:0/3096195648 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f04005040 con 0x7f0f080f6510 2023-10-12T22:34:05.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.982+0000 7f0f0d4be700 1 --2- 172.21.15.195:0/3096195648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f080f6510 0x7f0f08106730 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f0f04013510 tx=0x7f0f0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:05.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.982+0000 7f0efdffb700 1 -- 172.21.15.195:0/3096195648 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f04009320 con 0x7f0f080f6510 2023-10-12T22:34:05.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.982+0000 7f0f0f722700 1 -- 172.21.15.195:0/3096195648 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f08106e20 con 0x7f0f080f6510 2023-10-12T22:34:05.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.982+0000 7f0efdffb700 1 -- 172.21.15.195:0/3096195648 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f04013f00 con 0x7f0f080f6510 2023-10-12T22:34:05.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.982+0000 7f0efdffb700 1 -- 172.21.15.195:0/3096195648 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f04014d90 con 0x7f0f080f6510 2023-10-12T22:34:05.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.982+0000 7f0f0f722700 1 -- 172.21.15.195:0/3096195648 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f081072b0 con 0x7f0f080f6510 2023-10-12T22:34:05.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.983+0000 7f0efdffb700 1 -- 172.21.15.195:0/3096195648 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f040095f0 con 0x7f0f080f6510 2023-10-12T22:34:05.989 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.983+0000 7f0f0f722700 1 -- 172.21.15.195:0/3096195648 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f08045ad0 con 0x7f0f080f6510 2023-10-12T22:34:05.989 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.983+0000 7f0efdffb700 1 --2- 172.21.15.195:0/3096195648 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0ef402ff60 0x7f0ef4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:05.989 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.983+0000 7f0efdffb700 1 -- 172.21.15.195:0/3096195648 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f04041000 con 0x7f0f080f6510 2023-10-12T22:34:05.989 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.983+0000 7f0f0ccbd700 1 --2- 172.21.15.195:0/3096195648 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0ef402ff60 0x7f0ef4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:05.989 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.984+0000 7f0f0ccbd700 1 --2- 172.21.15.195:0/3096195648 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0ef402ff60 0x7f0ef4032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0ef80096a0 tx=0x7f0ef8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:05.990 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:05.988+0000 7f0efdffb700 1 -- 172.21.15.195:0/3096195648 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f0401d7f0 con 0x7f0f080f6510 2023-10-12T22:34:06.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:06.194+0000 7f0f0f722700 1 -- 172.21.15.195:0/3096195648 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f080fbf60 con 0x7f0f080f6510 2023-10-12T22:34:06.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:06.195+0000 7f0efdffb700 1 -- 172.21.15.195:0/3096195648 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0f04010230 con 0x7f0f080f6510 2023-10-12T22:34:06.196 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:06.197 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:06.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:06.196+0000 7f0f0f722700 1 -- 172.21.15.195:0/3096195648 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0ef402ff60 msgr2=0x7f0ef4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:06.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:06.196+0000 7f0f0f722700 1 --2- 172.21.15.195:0/3096195648 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0ef402ff60 0x7f0ef4032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0ef80096a0 tx=0x7f0ef8006b40).stop 2023-10-12T22:34:06.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:06.196+0000 7f0f0f722700 1 -- 172.21.15.195:0/3096195648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f080f6510 msgr2=0x7f0f08106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:06.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:06.196+0000 7f0f0f722700 1 --2- 172.21.15.195:0/3096195648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f080f6510 0x7f0f08106730 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f0f04013510 tx=0x7f0f0400b040).stop 2023-10-12T22:34:06.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:06.196+0000 7f0f0f722700 1 -- 172.21.15.195:0/3096195648 shutdown_connections 2023-10-12T22:34:06.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:06.196+0000 7f0f0f722700 1 --2- 172.21.15.195:0/3096195648 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0ef402ff60 0x7f0ef4032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:06.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:06.196+0000 7f0f0f722700 1 --2- 172.21.15.195:0/3096195648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f080f6510 0x7f0f08106730 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:06.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:06.196+0000 7f0f0f722700 1 -- 172.21.15.195:0/3096195648 >> 172.21.15.195:0/3096195648 conn(0x7f0f080f1c70 msgr2=0x7f0f080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:06.200 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:06.196+0000 7f0f0f722700 1 -- 172.21.15.195:0/3096195648 shutdown_connections 2023-10-12T22:34:06.200 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:06.196+0000 7f0f0f722700 1 -- 172.21.15.195:0/3096195648 wait complete. 2023-10-12T22:34:06.201 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:06.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:04.821404+0000 mgr.smithi131.efktma (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:07.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:06.196163+0000 mon.smithi131 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.195:0/3096195648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:07.574 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:07.574 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:08.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:06.821731+0000 mgr.smithi131.efktma (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:08.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.741+0000 7fbbd49d5700 1 -- 172.21.15.195:0/2819634102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbd00f5330 msgr2=0x7fbbd00f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:08.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.741+0000 7fbbd49d5700 1 --2- 172.21.15.195:0/2819634102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbd00f5330 0x7fbbd00f5750 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fbbb8009fb0 tx=0x7fbbb8009b30).stop 2023-10-12T22:34:08.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.741+0000 7fbbd49d5700 1 -- 172.21.15.195:0/2819634102 shutdown_connections 2023-10-12T22:34:08.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.741+0000 7fbbd49d5700 1 --2- 172.21.15.195:0/2819634102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbd00f5330 0x7fbbd00f5750 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:08.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.741+0000 7fbbd49d5700 1 -- 172.21.15.195:0/2819634102 >> 172.21.15.195:0/2819634102 conn(0x7fbbd00f0a90 msgr2=0x7fbbd00f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:08.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.741+0000 7fbbd49d5700 1 -- 172.21.15.195:0/2819634102 shutdown_connections 2023-10-12T22:34:08.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.741+0000 7fbbd49d5700 1 -- 172.21.15.195:0/2819634102 wait complete. 2023-10-12T22:34:08.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.741+0000 7fbbd49d5700 1 Processor -- start 2023-10-12T22:34:08.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.742+0000 7fbbd49d5700 1 -- start start 2023-10-12T22:34:08.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.742+0000 7fbbd49d5700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbd00f5330 0x7fbbd01058d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:08.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.742+0000 7fbbd49d5700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd0105dc0 con 0x7fbbd00f5330 2023-10-12T22:34:08.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.742+0000 7fbbce59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbd00f5330 0x7fbbd01058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:08.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.742+0000 7fbbce59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbd00f5330 0x7fbbd01058d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:40614/0 (socket says 172.21.15.195:40614) 2023-10-12T22:34:08.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.742+0000 7fbbce59c700 1 -- 172.21.15.195:0/3966396329 learned_addr learned my addr 172.21.15.195:0/3966396329 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:08.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.742+0000 7fbbce59c700 1 -- 172.21.15.195:0/3966396329 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8005040 con 0x7fbbd00f5330 2023-10-12T22:34:08.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.743+0000 7fbbce59c700 1 --2- 172.21.15.195:0/3966396329 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbd00f5330 0x7fbbd01058d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fbbb8014000 tx=0x7fbbb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:08.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.743+0000 7fbbc6ffd700 1 -- 172.21.15.195:0/3966396329 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbb80095f0 con 0x7fbbd00f5330 2023-10-12T22:34:08.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.743+0000 7fbbc6ffd700 1 -- 172.21.15.195:0/3966396329 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbb8007e70 con 0x7fbbd00f5330 2023-10-12T22:34:08.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.743+0000 7fbbc6ffd700 1 -- 172.21.15.195:0/3966396329 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbb8014ea0 con 0x7fbbd00f5330 2023-10-12T22:34:08.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.743+0000 7fbbd49d5700 1 -- 172.21.15.195:0/3966396329 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd0105fc0 con 0x7fbbd00f5330 2023-10-12T22:34:08.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.743+0000 7fbbd49d5700 1 -- 172.21.15.195:0/3966396329 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd0106450 con 0x7fbbd00f5330 2023-10-12T22:34:08.750 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.744+0000 7fbbc6ffd700 1 -- 172.21.15.195:0/3966396329 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbbb8025070 con 0x7fbbd00f5330 2023-10-12T22:34:08.750 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.744+0000 7fbbd49d5700 1 -- 172.21.15.195:0/3966396329 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd0045ad0 con 0x7fbbd00f5330 2023-10-12T22:34:08.750 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.744+0000 7fbbc6ffd700 1 --2- 172.21.15.195:0/3966396329 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbbbc02ff60 0x7fbbbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:08.751 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.744+0000 7fbbc6ffd700 1 -- 172.21.15.195:0/3966396329 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbbb8040a40 con 0x7fbbd00f5330 2023-10-12T22:34:08.751 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.744+0000 7fbbcdd9b700 1 --2- 172.21.15.195:0/3966396329 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbbbc02ff60 0x7fbbbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:08.751 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.744+0000 7fbbcdd9b700 1 --2- 172.21.15.195:0/3966396329 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbbbc02ff60 0x7fbbbc032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fbbc00096a0 tx=0x7fbbc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:08.751 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.748+0000 7fbbc6ffd700 1 -- 172.21.15.195:0/3966396329 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbbb80079c0 con 0x7fbbd00f5330 2023-10-12T22:34:08.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.955+0000 7fbbd49d5700 1 -- 172.21.15.195:0/3966396329 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbd005ea50 con 0x7fbbd00f5330 2023-10-12T22:34:08.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.956+0000 7fbbc6ffd700 1 -- 172.21.15.195:0/3966396329 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbbb800f300 con 0x7fbbd00f5330 2023-10-12T22:34:08.958 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:08.958 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:08.959 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.957+0000 7fbbd49d5700 1 -- 172.21.15.195:0/3966396329 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbbbc02ff60 msgr2=0x7fbbbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:08.960 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.957+0000 7fbbd49d5700 1 --2- 172.21.15.195:0/3966396329 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbbbc02ff60 0x7fbbbc032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fbbc00096a0 tx=0x7fbbc0006b40).stop 2023-10-12T22:34:08.960 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.957+0000 7fbbd49d5700 1 -- 172.21.15.195:0/3966396329 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbd00f5330 msgr2=0x7fbbd01058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:08.960 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.958+0000 7fbbd49d5700 1 --2- 172.21.15.195:0/3966396329 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbd00f5330 0x7fbbd01058d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fbbb8014000 tx=0x7fbbb800b040).stop 2023-10-12T22:34:08.960 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.958+0000 7fbbd49d5700 1 -- 172.21.15.195:0/3966396329 shutdown_connections 2023-10-12T22:34:08.960 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.958+0000 7fbbd49d5700 1 --2- 172.21.15.195:0/3966396329 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbbbc02ff60 0x7fbbbc032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:08.961 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.958+0000 7fbbd49d5700 1 --2- 172.21.15.195:0/3966396329 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbd00f5330 0x7fbbd01058d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:08.961 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.958+0000 7fbbd49d5700 1 -- 172.21.15.195:0/3966396329 >> 172.21.15.195:0/3966396329 conn(0x7fbbd00f0a90 msgr2=0x7fbbd00f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:08.961 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.958+0000 7fbbd49d5700 1 -- 172.21.15.195:0/3966396329 shutdown_connections 2023-10-12T22:34:08.961 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:08.958+0000 7fbbd49d5700 1 -- 172.21.15.195:0/3966396329 wait complete. 2023-10-12T22:34:08.962 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:09.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:09 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:08.957536+0000 mon.smithi131 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.195:0/3966396329' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:10.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:10.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:10.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:08.822100+0000 mgr.smithi131.efktma (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:11.437 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.435+0000 7fb147a56700 1 -- 172.21.15.195:0/246426473 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1400f5410 msgr2=0x7fb1400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:11.438 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.435+0000 7fb147a56700 1 --2- 172.21.15.195:0/246426473 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1400f5410 0x7fb1400f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fb138009fb0 tx=0x7fb138009b30).stop 2023-10-12T22:34:11.438 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.435+0000 7fb147a56700 1 -- 172.21.15.195:0/246426473 shutdown_connections 2023-10-12T22:34:11.438 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.435+0000 7fb147a56700 1 --2- 172.21.15.195:0/246426473 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1400f5410 0x7fb1400f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:11.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.435+0000 7fb147a56700 1 -- 172.21.15.195:0/246426473 >> 172.21.15.195:0/246426473 conn(0x7fb1400f0b70 msgr2=0x7fb1400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:11.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.435+0000 7fb147a56700 1 -- 172.21.15.195:0/246426473 shutdown_connections 2023-10-12T22:34:11.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.435+0000 7fb147a56700 1 -- 172.21.15.195:0/246426473 wait complete. 2023-10-12T22:34:11.440 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.436+0000 7fb147a56700 1 Processor -- start 2023-10-12T22:34:11.440 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.436+0000 7fb147a56700 1 -- start start 2023-10-12T22:34:11.440 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.436+0000 7fb147a56700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1400f5410 0x7fb140105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:11.440 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.436+0000 7fb147a56700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb140105b20 con 0x7fb1400f5410 2023-10-12T22:34:11.442 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.437+0000 7fb1457f2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1400f5410 0x7fb140105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:11.443 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.437+0000 7fb1457f2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1400f5410 0x7fb140105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:40634/0 (socket says 172.21.15.195:40634) 2023-10-12T22:34:11.443 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.437+0000 7fb1457f2700 1 -- 172.21.15.195:0/2184721041 learned_addr learned my addr 172.21.15.195:0/2184721041 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:11.443 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.437+0000 7fb1457f2700 1 -- 172.21.15.195:0/2184721041 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb138005040 con 0x7fb1400f5410 2023-10-12T22:34:11.443 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.437+0000 7fb1457f2700 1 --2- 172.21.15.195:0/2184721041 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1400f5410 0x7fb140105630 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fb138013510 tx=0x7fb13800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:11.444 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.437+0000 7fb13dffb700 1 -- 172.21.15.195:0/2184721041 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb138009320 con 0x7fb1400f5410 2023-10-12T22:34:11.444 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.437+0000 7fb147a56700 1 -- 172.21.15.195:0/2184721041 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb140105d20 con 0x7fb1400f5410 2023-10-12T22:34:11.444 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.437+0000 7fb13dffb700 1 -- 172.21.15.195:0/2184721041 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb138013f00 con 0x7fb1400f5410 2023-10-12T22:34:11.445 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.437+0000 7fb13dffb700 1 -- 172.21.15.195:0/2184721041 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb138014d90 con 0x7fb1400f5410 2023-10-12T22:34:11.445 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.437+0000 7fb147a56700 1 -- 172.21.15.195:0/2184721041 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1401061b0 con 0x7fb1400f5410 2023-10-12T22:34:11.445 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.438+0000 7fb13dffb700 1 -- 172.21.15.195:0/2184721041 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1380095f0 con 0x7fb1400f5410 2023-10-12T22:34:11.446 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.438+0000 7fb147a56700 1 -- 172.21.15.195:0/2184721041 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb140045ad0 con 0x7fb1400f5410 2023-10-12T22:34:11.446 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.438+0000 7fb13dffb700 1 --2- 172.21.15.195:0/2184721041 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb12802ff60 0x7fb128032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:11.447 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.438+0000 7fb13dffb700 1 -- 172.21.15.195:0/2184721041 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb138040fc0 con 0x7fb1400f5410 2023-10-12T22:34:11.447 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.438+0000 7fb144ff1700 1 --2- 172.21.15.195:0/2184721041 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb12802ff60 0x7fb128032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:11.447 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.439+0000 7fb144ff1700 1 --2- 172.21.15.195:0/2184721041 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb12802ff60 0x7fb128032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb1340096a0 tx=0x7fb134006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:11.448 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.443+0000 7fb13dffb700 1 -- 172.21.15.195:0/2184721041 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb138014420 con 0x7fb1400f5410 2023-10-12T22:34:11.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.646+0000 7fb147a56700 1 -- 172.21.15.195:0/2184721041 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1400fae60 con 0x7fb1400f5410 2023-10-12T22:34:11.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.648+0000 7fb13dffb700 1 -- 172.21.15.195:0/2184721041 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb138010230 con 0x7fb1400f5410 2023-10-12T22:34:11.648 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:11.649 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:11.651 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.649+0000 7fb147a56700 1 -- 172.21.15.195:0/2184721041 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb12802ff60 msgr2=0x7fb128032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:11.651 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.649+0000 7fb147a56700 1 --2- 172.21.15.195:0/2184721041 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb12802ff60 0x7fb128032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb1340096a0 tx=0x7fb134006b40).stop 2023-10-12T22:34:11.651 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.649+0000 7fb147a56700 1 -- 172.21.15.195:0/2184721041 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1400f5410 msgr2=0x7fb140105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:11.652 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.649+0000 7fb147a56700 1 --2- 172.21.15.195:0/2184721041 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1400f5410 0x7fb140105630 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fb138013510 tx=0x7fb13800b040).stop 2023-10-12T22:34:11.652 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.649+0000 7fb147a56700 1 -- 172.21.15.195:0/2184721041 shutdown_connections 2023-10-12T22:34:11.652 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.649+0000 7fb147a56700 1 --2- 172.21.15.195:0/2184721041 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb12802ff60 0x7fb128032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:11.652 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.649+0000 7fb147a56700 1 --2- 172.21.15.195:0/2184721041 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1400f5410 0x7fb140105630 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:11.652 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.649+0000 7fb147a56700 1 -- 172.21.15.195:0/2184721041 >> 172.21.15.195:0/2184721041 conn(0x7fb1400f0b70 msgr2=0x7fb1400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:11.653 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.649+0000 7fb147a56700 1 -- 172.21.15.195:0/2184721041 shutdown_connections 2023-10-12T22:34:11.653 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:11.649+0000 7fb147a56700 1 -- 172.21.15.195:0/2184721041 wait complete. 2023-10-12T22:34:11.653 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:12.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:10.822385+0000 mgr.smithi131.efktma (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:12.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:11.648471+0000 mon.smithi131 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.195:0/2184721041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:13.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:13.027 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:14.192 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.189+0000 7f63d0bd7700 1 -- 172.21.15.195:0/2417516533 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63cc0f5410 msgr2=0x7f63cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:14.192 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.189+0000 7f63d0bd7700 1 --2- 172.21.15.195:0/2417516533 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63cc0f5410 0x7f63cc0f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f63b4009fb0 tx=0x7f63b4009b30).stop 2023-10-12T22:34:14.193 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.189+0000 7f63d0bd7700 1 -- 172.21.15.195:0/2417516533 shutdown_connections 2023-10-12T22:34:14.193 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.189+0000 7f63d0bd7700 1 --2- 172.21.15.195:0/2417516533 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63cc0f5410 0x7f63cc0f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:14.193 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.189+0000 7f63d0bd7700 1 -- 172.21.15.195:0/2417516533 >> 172.21.15.195:0/2417516533 conn(0x7f63cc0f0b70 msgr2=0x7f63cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:14.193 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.189+0000 7f63d0bd7700 1 -- 172.21.15.195:0/2417516533 shutdown_connections 2023-10-12T22:34:14.193 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.189+0000 7f63d0bd7700 1 -- 172.21.15.195:0/2417516533 wait complete. 2023-10-12T22:34:14.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.190+0000 7f63d0bd7700 1 Processor -- start 2023-10-12T22:34:14.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.190+0000 7f63d0bd7700 1 -- start start 2023-10-12T22:34:14.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.190+0000 7f63d0bd7700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63cc0f5410 0x7f63cc105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:14.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.190+0000 7f63d0bd7700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc105af0 con 0x7f63cc0f5410 2023-10-12T22:34:14.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.190+0000 7f63ca59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63cc0f5410 0x7f63cc105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:14.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.190+0000 7f63ca59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63cc0f5410 0x7f63cc105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:40666/0 (socket says 172.21.15.195:40666) 2023-10-12T22:34:14.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.190+0000 7f63ca59c700 1 -- 172.21.15.195:0/3355376008 learned_addr learned my addr 172.21.15.195:0/3355376008 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:14.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.191+0000 7f63ca59c700 1 -- 172.21.15.195:0/3355376008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b4005040 con 0x7f63cc0f5410 2023-10-12T22:34:14.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.191+0000 7f63ca59c700 1 --2- 172.21.15.195:0/3355376008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63cc0f5410 0x7f63cc105600 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f63b4013510 tx=0x7f63b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:14.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.191+0000 7f63c2ffd700 1 -- 172.21.15.195:0/3355376008 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b4009320 con 0x7f63cc0f5410 2023-10-12T22:34:14.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.191+0000 7f63d0bd7700 1 -- 172.21.15.195:0/3355376008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc105cf0 con 0x7f63cc0f5410 2023-10-12T22:34:14.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.191+0000 7f63c2ffd700 1 -- 172.21.15.195:0/3355376008 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63b4013f00 con 0x7f63cc0f5410 2023-10-12T22:34:14.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.191+0000 7f63c2ffd700 1 -- 172.21.15.195:0/3355376008 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b4014d90 con 0x7f63cc0f5410 2023-10-12T22:34:14.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.191+0000 7f63d0bd7700 1 -- 172.21.15.195:0/3355376008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc106180 con 0x7f63cc0f5410 2023-10-12T22:34:14.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.192+0000 7f63c2ffd700 1 -- 172.21.15.195:0/3355376008 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63b40095f0 con 0x7f63cc0f5410 2023-10-12T22:34:14.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.192+0000 7f63d0bd7700 1 -- 172.21.15.195:0/3355376008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc045ad0 con 0x7f63cc0f5410 2023-10-12T22:34:14.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.192+0000 7f63c2ffd700 1 --2- 172.21.15.195:0/3355376008 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f63b802ff60 0x7f63b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:14.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.192+0000 7f63c2ffd700 1 -- 172.21.15.195:0/3355376008 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63b4041000 con 0x7f63cc0f5410 2023-10-12T22:34:14.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.192+0000 7f63c9d9b700 1 --2- 172.21.15.195:0/3355376008 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f63b802ff60 0x7f63b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:14.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.193+0000 7f63c9d9b700 1 --2- 172.21.15.195:0/3355376008 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f63b802ff60 0x7f63b8032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f63bc0096a0 tx=0x7f63bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:14.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.197+0000 7f63c2ffd700 1 -- 172.21.15.195:0/3355376008 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63b4014420 con 0x7f63cc0f5410 2023-10-12T22:34:14.403 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.402+0000 7f63d0bd7700 1 -- 172.21.15.195:0/3355376008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63cc0fae60 con 0x7f63cc0f5410 2023-10-12T22:34:14.404 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.403+0000 7f63c2ffd700 1 -- 172.21.15.195:0/3355376008 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f63b4010230 con 0x7f63cc0f5410 2023-10-12T22:34:14.404 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:14.405 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:14.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.404+0000 7f63d0bd7700 1 -- 172.21.15.195:0/3355376008 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f63b802ff60 msgr2=0x7f63b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:14.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.404+0000 7f63d0bd7700 1 --2- 172.21.15.195:0/3355376008 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f63b802ff60 0x7f63b8032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f63bc0096a0 tx=0x7f63bc006b40).stop 2023-10-12T22:34:14.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.404+0000 7f63d0bd7700 1 -- 172.21.15.195:0/3355376008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63cc0f5410 msgr2=0x7f63cc105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:14.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.404+0000 7f63d0bd7700 1 --2- 172.21.15.195:0/3355376008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63cc0f5410 0x7f63cc105600 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f63b4013510 tx=0x7f63b400b040).stop 2023-10-12T22:34:14.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.405+0000 7f63d0bd7700 1 -- 172.21.15.195:0/3355376008 shutdown_connections 2023-10-12T22:34:14.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.405+0000 7f63d0bd7700 1 --2- 172.21.15.195:0/3355376008 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f63b802ff60 0x7f63b8032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:14.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.405+0000 7f63d0bd7700 1 --2- 172.21.15.195:0/3355376008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63cc0f5410 0x7f63cc105600 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:14.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.405+0000 7f63d0bd7700 1 -- 172.21.15.195:0/3355376008 >> 172.21.15.195:0/3355376008 conn(0x7f63cc0f0b70 msgr2=0x7f63cc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:14.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.405+0000 7f63d0bd7700 1 -- 172.21.15.195:0/3355376008 shutdown_connections 2023-10-12T22:34:14.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:14.405+0000 7f63d0bd7700 1 -- 172.21.15.195:0/3355376008 wait complete. 2023-10-12T22:34:14.408 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:14.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:14 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:12.822741+0000 mgr.smithi131.efktma (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:15.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:15 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:14.404351+0000 mon.smithi131 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.195:0/3355376008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:15.815 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:15.815 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:16.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:16 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:14.823157+0000 mgr.smithi131.efktma (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:16.857 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.854+0000 7f76bbaa3700 1 -- 172.21.15.195:0/1848420697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76b40f6510 msgr2=0x7f76b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:16.857 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.854+0000 7f76bbaa3700 1 --2- 172.21.15.195:0/1848420697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76b40f6510 0x7f76b40f6930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f76a4009fb0 tx=0x7f76a4009b30).stop 2023-10-12T22:34:16.858 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.854+0000 7f76bbaa3700 1 -- 172.21.15.195:0/1848420697 shutdown_connections 2023-10-12T22:34:16.858 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.854+0000 7f76bbaa3700 1 --2- 172.21.15.195:0/1848420697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76b40f6510 0x7f76b40f6930 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:16.858 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.854+0000 7f76bbaa3700 1 -- 172.21.15.195:0/1848420697 >> 172.21.15.195:0/1848420697 conn(0x7f76b40f1c70 msgr2=0x7f76b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:16.858 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.854+0000 7f76bbaa3700 1 -- 172.21.15.195:0/1848420697 shutdown_connections 2023-10-12T22:34:16.858 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.854+0000 7f76bbaa3700 1 -- 172.21.15.195:0/1848420697 wait complete. 2023-10-12T22:34:16.859 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.855+0000 7f76bbaa3700 1 Processor -- start 2023-10-12T22:34:16.859 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.855+0000 7f76bbaa3700 1 -- start start 2023-10-12T22:34:16.859 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.855+0000 7f76bbaa3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76b40f6510 0x7f76b406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:16.859 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.855+0000 7f76bbaa3700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b4069e20 con 0x7f76b40f6510 2023-10-12T22:34:16.859 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.855+0000 7f76b983f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76b40f6510 0x7f76b406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:16.860 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.855+0000 7f76b983f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76b40f6510 0x7f76b406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:40678/0 (socket says 172.21.15.195:40678) 2023-10-12T22:34:16.860 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.855+0000 7f76b983f700 1 -- 172.21.15.195:0/3859977831 learned_addr learned my addr 172.21.15.195:0/3859977831 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:16.860 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.855+0000 7f76b983f700 1 -- 172.21.15.195:0/3859977831 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a4005040 con 0x7f76b40f6510 2023-10-12T22:34:16.860 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.856+0000 7f76b983f700 1 --2- 172.21.15.195:0/3859977831 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76b40f6510 0x7f76b406b9a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f76a4013510 tx=0x7f76a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:16.860 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.856+0000 7f76aa7fc700 1 -- 172.21.15.195:0/3859977831 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76a4009320 con 0x7f76b40f6510 2023-10-12T22:34:16.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.856+0000 7f76bbaa3700 1 -- 172.21.15.195:0/3859977831 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b4069fc0 con 0x7f76b40f6510 2023-10-12T22:34:16.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.856+0000 7f76aa7fc700 1 -- 172.21.15.195:0/3859977831 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76a4013f00 con 0x7f76b40f6510 2023-10-12T22:34:16.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.856+0000 7f76aa7fc700 1 -- 172.21.15.195:0/3859977831 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76a4014d90 con 0x7f76b40f6510 2023-10-12T22:34:16.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.856+0000 7f76bbaa3700 1 -- 172.21.15.195:0/3859977831 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b406a450 con 0x7f76b40f6510 2023-10-12T22:34:16.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.857+0000 7f76aa7fc700 1 -- 172.21.15.195:0/3859977831 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76a40095f0 con 0x7f76b40f6510 2023-10-12T22:34:16.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.857+0000 7f76bbaa3700 1 -- 172.21.15.195:0/3859977831 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b4059100 con 0x7f76b40f6510 2023-10-12T22:34:16.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.857+0000 7f76aa7fc700 1 --2- 172.21.15.195:0/3859977831 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f76a002ff60 0x7f76a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:16.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.857+0000 7f76aa7fc700 1 -- 172.21.15.195:0/3859977831 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76a4041000 con 0x7f76b40f6510 2023-10-12T22:34:16.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.857+0000 7f76b903e700 1 --2- 172.21.15.195:0/3859977831 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f76a002ff60 0x7f76a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:16.864 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.857+0000 7f76b903e700 1 --2- 172.21.15.195:0/3859977831 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f76a002ff60 0x7f76a0032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f76b00096a0 tx=0x7f76b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:16.864 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:16.861+0000 7f76aa7fc700 1 -- 172.21.15.195:0/3859977831 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76a401d7f0 con 0x7f76b40f6510 2023-10-12T22:34:17.069 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:17.067+0000 7f76bbaa3700 1 -- 172.21.15.195:0/3859977831 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76b40fbf60 con 0x7f76b40f6510 2023-10-12T22:34:17.069 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:17.068+0000 7f76aa7fc700 1 -- 172.21.15.195:0/3859977831 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f76a4010230 con 0x7f76b40f6510 2023-10-12T22:34:17.070 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:17.070 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:17.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:17.070+0000 7f76bbaa3700 1 -- 172.21.15.195:0/3859977831 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f76a002ff60 msgr2=0x7f76a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:17.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:17.070+0000 7f76bbaa3700 1 --2- 172.21.15.195:0/3859977831 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f76a002ff60 0x7f76a0032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f76b00096a0 tx=0x7f76b0006b40).stop 2023-10-12T22:34:17.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:17.070+0000 7f76bbaa3700 1 -- 172.21.15.195:0/3859977831 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76b40f6510 msgr2=0x7f76b406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:17.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:17.070+0000 7f76bbaa3700 1 --2- 172.21.15.195:0/3859977831 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76b40f6510 0x7f76b406b9a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f76a4013510 tx=0x7f76a400b040).stop 2023-10-12T22:34:17.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:17.070+0000 7f76bbaa3700 1 -- 172.21.15.195:0/3859977831 shutdown_connections 2023-10-12T22:34:17.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:17.070+0000 7f76bbaa3700 1 --2- 172.21.15.195:0/3859977831 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f76a002ff60 0x7f76a0032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:17.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:17.070+0000 7f76bbaa3700 1 --2- 172.21.15.195:0/3859977831 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76b40f6510 0x7f76b406b9a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:17.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:17.070+0000 7f76bbaa3700 1 -- 172.21.15.195:0/3859977831 >> 172.21.15.195:0/3859977831 conn(0x7f76b40f1c70 msgr2=0x7f76b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:17.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:17.070+0000 7f76bbaa3700 1 -- 172.21.15.195:0/3859977831 shutdown_connections 2023-10-12T22:34:17.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:17.070+0000 7f76bbaa3700 1 -- 172.21.15.195:0/3859977831 wait complete. 2023-10-12T22:34:17.073 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:17.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:17 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:17.069435+0000 mon.smithi131 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.195:0/3859977831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:18.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:18.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:18.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:18 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:16.823497+0000 mgr.smithi131.efktma (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:19.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.611+0000 7f3178d1a700 1 -- 172.21.15.195:0/1248436387 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f31740f5410 msgr2=0x7f31740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:19.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.611+0000 7f3178d1a700 1 --2- 172.21.15.195:0/1248436387 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f31740f5410 0x7f31740f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f315c009fb0 tx=0x7f315c009b30).stop 2023-10-12T22:34:19.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.611+0000 7f3178d1a700 1 -- 172.21.15.195:0/1248436387 shutdown_connections 2023-10-12T22:34:19.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.611+0000 7f3178d1a700 1 --2- 172.21.15.195:0/1248436387 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f31740f5410 0x7f31740f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:19.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.611+0000 7f3178d1a700 1 -- 172.21.15.195:0/1248436387 >> 172.21.15.195:0/1248436387 conn(0x7f31740f0b70 msgr2=0x7f31740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:19.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.611+0000 7f3178d1a700 1 -- 172.21.15.195:0/1248436387 shutdown_connections 2023-10-12T22:34:19.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.612+0000 7f3178d1a700 1 -- 172.21.15.195:0/1248436387 wait complete. 2023-10-12T22:34:19.616 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.612+0000 7f3178d1a700 1 Processor -- start 2023-10-12T22:34:19.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.612+0000 7f3178d1a700 1 -- start start 2023-10-12T22:34:19.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.612+0000 7f3178d1a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f31740f5410 0x7f3174105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:19.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.612+0000 7f3178d1a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3174105b20 con 0x7f31740f5410 2023-10-12T22:34:19.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.613+0000 7f317259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f31740f5410 0x7f3174105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:19.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.613+0000 7f317259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f31740f5410 0x7f3174105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:33522/0 (socket says 172.21.15.195:33522) 2023-10-12T22:34:19.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.613+0000 7f317259c700 1 -- 172.21.15.195:0/2418882985 learned_addr learned my addr 172.21.15.195:0/2418882985 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:19.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.613+0000 7f317259c700 1 -- 172.21.15.195:0/2418882985 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c005040 con 0x7f31740f5410 2023-10-12T22:34:19.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.613+0000 7f317259c700 1 --2- 172.21.15.195:0/2418882985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f31740f5410 0x7f3174105630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f315c013510 tx=0x7f315c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:19.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.613+0000 7f316affd700 1 -- 172.21.15.195:0/2418882985 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f315c009320 con 0x7f31740f5410 2023-10-12T22:34:19.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.613+0000 7f3178d1a700 1 -- 172.21.15.195:0/2418882985 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3174105d20 con 0x7f31740f5410 2023-10-12T22:34:19.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.613+0000 7f316affd700 1 -- 172.21.15.195:0/2418882985 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f315c013f00 con 0x7f31740f5410 2023-10-12T22:34:19.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.613+0000 7f316affd700 1 -- 172.21.15.195:0/2418882985 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f315c014d90 con 0x7f31740f5410 2023-10-12T22:34:19.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.613+0000 7f3178d1a700 1 -- 172.21.15.195:0/2418882985 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31741061b0 con 0x7f31740f5410 2023-10-12T22:34:19.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.614+0000 7f316affd700 1 -- 172.21.15.195:0/2418882985 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f315c0263a0 con 0x7f31740f5410 2023-10-12T22:34:19.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.614+0000 7f316affd700 1 --2- 172.21.15.195:0/2418882985 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f316002ff60 0x7f3160032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:19.620 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.614+0000 7f316affd700 1 -- 172.21.15.195:0/2418882985 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f315c040fe0 con 0x7f31740f5410 2023-10-12T22:34:19.620 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.614+0000 7f3178d1a700 1 -- 172.21.15.195:0/2418882985 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3174045ad0 con 0x7f31740f5410 2023-10-12T22:34:19.620 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.615+0000 7f3171d9b700 1 --2- 172.21.15.195:0/2418882985 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f316002ff60 0x7f3160032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:19.620 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.615+0000 7f3171d9b700 1 --2- 172.21.15.195:0/2418882985 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f316002ff60 0x7f3160032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f31640096a0 tx=0x7f3164006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:19.621 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.619+0000 7f316affd700 1 -- 172.21.15.195:0/2418882985 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f315c012920 con 0x7f31740f5410 2023-10-12T22:34:19.824 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.823+0000 7f3178d1a700 1 -- 172.21.15.195:0/2418882985 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3174106ed0 con 0x7f31740f5410 2023-10-12T22:34:19.825 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.824+0000 7f316affd700 1 -- 172.21.15.195:0/2418882985 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f315c02b020 con 0x7f31740f5410 2023-10-12T22:34:19.826 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:19.826 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:19.827 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.825+0000 7f3178d1a700 1 -- 172.21.15.195:0/2418882985 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f316002ff60 msgr2=0x7f3160032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:19.827 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.825+0000 7f3178d1a700 1 --2- 172.21.15.195:0/2418882985 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f316002ff60 0x7f3160032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f31640096a0 tx=0x7f3164006b40).stop 2023-10-12T22:34:19.827 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.825+0000 7f3178d1a700 1 -- 172.21.15.195:0/2418882985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f31740f5410 msgr2=0x7f3174105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:19.827 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.825+0000 7f3178d1a700 1 --2- 172.21.15.195:0/2418882985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f31740f5410 0x7f3174105630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f315c013510 tx=0x7f315c00b040).stop 2023-10-12T22:34:19.828 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.825+0000 7f3178d1a700 1 -- 172.21.15.195:0/2418882985 shutdown_connections 2023-10-12T22:34:19.828 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.825+0000 7f3178d1a700 1 --2- 172.21.15.195:0/2418882985 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f316002ff60 0x7f3160032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:19.828 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.826+0000 7f3178d1a700 1 --2- 172.21.15.195:0/2418882985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f31740f5410 0x7f3174105630 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:19.828 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.826+0000 7f3178d1a700 1 -- 172.21.15.195:0/2418882985 >> 172.21.15.195:0/2418882985 conn(0x7f31740f0b70 msgr2=0x7f31740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:19.828 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.826+0000 7f3178d1a700 1 -- 172.21.15.195:0/2418882985 shutdown_connections 2023-10-12T22:34:19.829 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:19.826+0000 7f3178d1a700 1 -- 172.21.15.195:0/2418882985 wait complete. 2023-10-12T22:34:19.829 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:20.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:20 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:18.823884+0000 mgr.smithi131.efktma (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:20.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:20 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:19.825469+0000 mon.smithi131 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.195:0/2418882985' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:21.241 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:21.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:22.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.325+0000 7f2691a40700 1 -- 172.21.15.195:0/1950820563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268c0f78e0 msgr2=0x7f268c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:22.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.325+0000 7f2691a40700 1 --2- 172.21.15.195:0/1950820563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268c0f78e0 0x7f268c0f7d00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f2680009fb0 tx=0x7f2680009b30).stop 2023-10-12T22:34:22.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.325+0000 7f2691a40700 1 -- 172.21.15.195:0/1950820563 shutdown_connections 2023-10-12T22:34:22.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.325+0000 7f2691a40700 1 --2- 172.21.15.195:0/1950820563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268c0f78e0 0x7f268c0f7d00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:22.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.325+0000 7f2691a40700 1 -- 172.21.15.195:0/1950820563 >> 172.21.15.195:0/1950820563 conn(0x7f268c0f0120 msgr2=0x7f268c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:22.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.325+0000 7f2691a40700 1 -- 172.21.15.195:0/1950820563 shutdown_connections 2023-10-12T22:34:22.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.325+0000 7f2691a40700 1 -- 172.21.15.195:0/1950820563 wait complete. 2023-10-12T22:34:22.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.325+0000 7f2691a40700 1 Processor -- start 2023-10-12T22:34:22.330 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.325+0000 7f2691a40700 1 -- start start 2023-10-12T22:34:22.330 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.326+0000 7f2691a40700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268c0f78e0 0x7f268c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:22.330 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.326+0000 7f2691a40700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c1069d0 con 0x7f268c0f78e0 2023-10-12T22:34:22.330 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.326+0000 7f2690a3e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268c0f78e0 0x7f268c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:22.330 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.326+0000 7f2690a3e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268c0f78e0 0x7f268c1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:33546/0 (socket says 172.21.15.195:33546) 2023-10-12T22:34:22.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.326+0000 7f2690a3e700 1 -- 172.21.15.195:0/1061410748 learned_addr learned my addr 172.21.15.195:0/1061410748 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:22.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.327+0000 7f2690a3e700 1 -- 172.21.15.195:0/1061410748 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680005040 con 0x7f268c0f78e0 2023-10-12T22:34:22.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.327+0000 7f2690a3e700 1 --2- 172.21.15.195:0/1061410748 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268c0f78e0 0x7f268c1064e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f2680013510 tx=0x7f268000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:22.332 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.327+0000 7f26897fa700 1 -- 172.21.15.195:0/1061410748 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2680009320 con 0x7f268c0f78e0 2023-10-12T22:34:22.333 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.327+0000 7f2691a40700 1 -- 172.21.15.195:0/1061410748 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268c106bd0 con 0x7f268c0f78e0 2023-10-12T22:34:22.333 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.327+0000 7f26897fa700 1 -- 172.21.15.195:0/1061410748 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2680013f00 con 0x7f268c0f78e0 2023-10-12T22:34:22.333 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.327+0000 7f26897fa700 1 -- 172.21.15.195:0/1061410748 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2680014dc0 con 0x7f268c0f78e0 2023-10-12T22:34:22.334 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.327+0000 7f2691a40700 1 -- 172.21.15.195:0/1061410748 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268c107060 con 0x7f268c0f78e0 2023-10-12T22:34:22.334 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.328+0000 7f26897fa700 1 -- 172.21.15.195:0/1061410748 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26800095f0 con 0x7f268c0f78e0 2023-10-12T22:34:22.334 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.328+0000 7f26897fa700 1 --2- 172.21.15.195:0/1061410748 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f267c02ff60 0x7f267c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:22.335 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.328+0000 7f2691a40700 1 -- 172.21.15.195:0/1061410748 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268c045ad0 con 0x7f268c0f78e0 2023-10-12T22:34:22.335 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.328+0000 7f26897fa700 1 -- 172.21.15.195:0/1061410748 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2680041000 con 0x7f268c0f78e0 2023-10-12T22:34:22.336 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.328+0000 7f268bfff700 1 --2- 172.21.15.195:0/1061410748 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f267c02ff60 0x7f267c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:22.336 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.329+0000 7f268bfff700 1 --2- 172.21.15.195:0/1061410748 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f267c02ff60 0x7f267c032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f26780096a0 tx=0x7f2678006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:22.337 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.333+0000 7f26897fa700 1 -- 172.21.15.195:0/1061410748 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2680014420 con 0x7f268c0f78e0 2023-10-12T22:34:22.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:20.824185+0000 mgr.smithi131.efktma (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:22.542 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.541+0000 7f2691a40700 1 -- 172.21.15.195:0/1061410748 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f268c107f10 con 0x7f268c0f78e0 2023-10-12T22:34:22.543 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.542+0000 7f26897fa700 1 -- 172.21.15.195:0/1061410748 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f268002b020 con 0x7f268c0f78e0 2023-10-12T22:34:22.544 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:22.544 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:22.545 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.544+0000 7f2691a40700 1 -- 172.21.15.195:0/1061410748 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f267c02ff60 msgr2=0x7f267c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:22.545 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.544+0000 7f2691a40700 1 --2- 172.21.15.195:0/1061410748 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f267c02ff60 0x7f267c032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f26780096a0 tx=0x7f2678006b40).stop 2023-10-12T22:34:22.545 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.544+0000 7f2691a40700 1 -- 172.21.15.195:0/1061410748 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268c0f78e0 msgr2=0x7f268c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:22.546 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.544+0000 7f2691a40700 1 --2- 172.21.15.195:0/1061410748 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268c0f78e0 0x7f268c1064e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f2680013510 tx=0x7f268000b040).stop 2023-10-12T22:34:22.546 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.544+0000 7f2691a40700 1 -- 172.21.15.195:0/1061410748 shutdown_connections 2023-10-12T22:34:22.546 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.544+0000 7f2691a40700 1 --2- 172.21.15.195:0/1061410748 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f267c02ff60 0x7f267c032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:22.546 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.544+0000 7f2691a40700 1 --2- 172.21.15.195:0/1061410748 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f268c0f78e0 0x7f268c1064e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:22.546 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.544+0000 7f2691a40700 1 -- 172.21.15.195:0/1061410748 >> 172.21.15.195:0/1061410748 conn(0x7f268c0f0120 msgr2=0x7f268c0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:22.547 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.544+0000 7f2691a40700 1 -- 172.21.15.195:0/1061410748 shutdown_connections 2023-10-12T22:34:22.547 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:22.544+0000 7f2691a40700 1 -- 172.21.15.195:0/1061410748 wait complete. 2023-10-12T22:34:22.547 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:23.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:22.543586+0000 mon.smithi131 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.195:0/1061410748' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:23.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:22.932426+0000 mon.smithi131 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/mirror_snapshot_schedule"}]: dispatch 2023-10-12T22:34:23.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:22.939670+0000 mon.smithi131 (mon.0) 176 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/trash_purge_schedule"}]: dispatch 2023-10-12T22:34:23.935 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:23.936 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:24.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:22.824495+0000 mgr.smithi131.efktma (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:25.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.041+0000 7ff2af6e0700 1 -- 172.21.15.195:0/3477127576 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2a80f6510 msgr2=0x7ff2a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:25.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.041+0000 7ff2af6e0700 1 --2- 172.21.15.195:0/3477127576 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2a80f6510 0x7ff2a80f6930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7ff2a4009fb0 tx=0x7ff2a4009b30).stop 2023-10-12T22:34:25.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.041+0000 7ff2af6e0700 1 -- 172.21.15.195:0/3477127576 shutdown_connections 2023-10-12T22:34:25.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.041+0000 7ff2af6e0700 1 --2- 172.21.15.195:0/3477127576 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2a80f6510 0x7ff2a80f6930 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:25.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.041+0000 7ff2af6e0700 1 -- 172.21.15.195:0/3477127576 >> 172.21.15.195:0/3477127576 conn(0x7ff2a80f1c70 msgr2=0x7ff2a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:25.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.041+0000 7ff2af6e0700 1 -- 172.21.15.195:0/3477127576 shutdown_connections 2023-10-12T22:34:25.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.041+0000 7ff2af6e0700 1 -- 172.21.15.195:0/3477127576 wait complete. 2023-10-12T22:34:25.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.042+0000 7ff2af6e0700 1 Processor -- start 2023-10-12T22:34:25.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.042+0000 7ff2af6e0700 1 -- start start 2023-10-12T22:34:25.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.042+0000 7ff2af6e0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2a80f6510 0x7ff2a8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:25.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.042+0000 7ff2af6e0700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a8106c20 con 0x7ff2a80f6510 2023-10-12T22:34:25.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.042+0000 7ff2ad47c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2a80f6510 0x7ff2a8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:25.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.042+0000 7ff2ad47c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2a80f6510 0x7ff2a8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:33562/0 (socket says 172.21.15.195:33562) 2023-10-12T22:34:25.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.042+0000 7ff2ad47c700 1 -- 172.21.15.195:0/139707437 learned_addr learned my addr 172.21.15.195:0/139707437 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:25.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.042+0000 7ff2ad47c700 1 -- 172.21.15.195:0/139707437 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a4005040 con 0x7ff2a80f6510 2023-10-12T22:34:25.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.043+0000 7ff2ad47c700 1 --2- 172.21.15.195:0/139707437 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2a80f6510 0x7ff2a8106730 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7ff2a4013510 tx=0x7ff2a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:25.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.043+0000 7ff29dffb700 1 -- 172.21.15.195:0/139707437 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a4009320 con 0x7ff2a80f6510 2023-10-12T22:34:25.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.043+0000 7ff2af6e0700 1 -- 172.21.15.195:0/139707437 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a8106e20 con 0x7ff2a80f6510 2023-10-12T22:34:25.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.043+0000 7ff29dffb700 1 -- 172.21.15.195:0/139707437 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2a4013f00 con 0x7ff2a80f6510 2023-10-12T22:34:25.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.043+0000 7ff29dffb700 1 -- 172.21.15.195:0/139707437 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a4014d90 con 0x7ff2a80f6510 2023-10-12T22:34:25.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.043+0000 7ff2af6e0700 1 -- 172.21.15.195:0/139707437 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a81072b0 con 0x7ff2a80f6510 2023-10-12T22:34:25.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.044+0000 7ff29dffb700 1 -- 172.21.15.195:0/139707437 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2a40095f0 con 0x7ff2a80f6510 2023-10-12T22:34:25.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.044+0000 7ff2af6e0700 1 -- 172.21.15.195:0/139707437 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a8045ad0 con 0x7ff2a80f6510 2023-10-12T22:34:25.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.044+0000 7ff29dffb700 1 --2- 172.21.15.195:0/139707437 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff29402ff60 0x7ff294032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:25.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.044+0000 7ff29dffb700 1 -- 172.21.15.195:0/139707437 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2a4041000 con 0x7ff2a80f6510 2023-10-12T22:34:25.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.044+0000 7ff2acc7b700 1 --2- 172.21.15.195:0/139707437 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff29402ff60 0x7ff294032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:25.050 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.045+0000 7ff2acc7b700 1 --2- 172.21.15.195:0/139707437 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff29402ff60 0x7ff294032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ff2980096a0 tx=0x7ff298006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:25.050 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.049+0000 7ff29dffb700 1 -- 172.21.15.195:0/139707437 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2a401d7f0 con 0x7ff2a80f6510 2023-10-12T22:34:25.257 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.256+0000 7ff2af6e0700 1 -- 172.21.15.195:0/139707437 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2a80fbf60 con 0x7ff2a80f6510 2023-10-12T22:34:25.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.258+0000 7ff29dffb700 1 -- 172.21.15.195:0/139707437 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff2a4010230 con 0x7ff2a80f6510 2023-10-12T22:34:25.258 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:25.259 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:25.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.258+0000 7ff2af6e0700 1 -- 172.21.15.195:0/139707437 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff29402ff60 msgr2=0x7ff294032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:25.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.258+0000 7ff2af6e0700 1 --2- 172.21.15.195:0/139707437 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff29402ff60 0x7ff294032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ff2980096a0 tx=0x7ff298006b40).stop 2023-10-12T22:34:25.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.258+0000 7ff2af6e0700 1 -- 172.21.15.195:0/139707437 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2a80f6510 msgr2=0x7ff2a8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:25.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.258+0000 7ff2af6e0700 1 --2- 172.21.15.195:0/139707437 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2a80f6510 0x7ff2a8106730 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7ff2a4013510 tx=0x7ff2a400b040).stop 2023-10-12T22:34:25.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.258+0000 7ff2af6e0700 1 -- 172.21.15.195:0/139707437 shutdown_connections 2023-10-12T22:34:25.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.258+0000 7ff2af6e0700 1 --2- 172.21.15.195:0/139707437 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff29402ff60 0x7ff294032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:25.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.258+0000 7ff2af6e0700 1 --2- 172.21.15.195:0/139707437 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2a80f6510 0x7ff2a8106730 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:25.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.258+0000 7ff2af6e0700 1 -- 172.21.15.195:0/139707437 >> 172.21.15.195:0/139707437 conn(0x7ff2a80f1c70 msgr2=0x7ff2a80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:25.263 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.258+0000 7ff2af6e0700 1 -- 172.21.15.195:0/139707437 shutdown_connections 2023-10-12T22:34:25.263 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:25.258+0000 7ff2af6e0700 1 -- 172.21.15.195:0/139707437 wait complete. 2023-10-12T22:34:25.263 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:26.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:26 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:24.824815+0000 mgr.smithi131.efktma (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:26.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:26 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:25.257965+0000 mon.smithi131 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.195:0/139707437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:26.674 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:26.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:27.858 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.857+0000 7f8d42dd7700 1 -- 172.21.15.195:0/4096146448 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d3c0f6510 msgr2=0x7f8d3c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:27.859 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.857+0000 7f8d42dd7700 1 --2- 172.21.15.195:0/4096146448 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c0f6930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f8d30009fb0 tx=0x7f8d30009b30).stop 2023-10-12T22:34:27.859 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.857+0000 7f8d42dd7700 1 -- 172.21.15.195:0/4096146448 shutdown_connections 2023-10-12T22:34:27.859 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.857+0000 7f8d42dd7700 1 --2- 172.21.15.195:0/4096146448 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c0f6930 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:27.860 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.857+0000 7f8d42dd7700 1 -- 172.21.15.195:0/4096146448 >> 172.21.15.195:0/4096146448 conn(0x7f8d3c0f1c70 msgr2=0x7f8d3c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:27.860 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.857+0000 7f8d42dd7700 1 -- 172.21.15.195:0/4096146448 shutdown_connections 2023-10-12T22:34:27.860 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.857+0000 7f8d42dd7700 1 -- 172.21.15.195:0/4096146448 wait complete. 2023-10-12T22:34:27.860 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.858+0000 7f8d42dd7700 1 Processor -- start 2023-10-12T22:34:27.860 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.858+0000 7f8d42dd7700 1 -- start start 2023-10-12T22:34:27.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.858+0000 7f8d42dd7700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:27.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.858+0000 7f8d42dd7700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c106c20 con 0x7f8d3c0f6510 2023-10-12T22:34:27.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.858+0000 7f8d40b73700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:27.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.858+0000 7f8d40b73700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:33582/0 (socket says 172.21.15.195:33582) 2023-10-12T22:34:27.862 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.858+0000 7f8d40b73700 1 -- 172.21.15.195:0/665384209 learned_addr learned my addr 172.21.15.195:0/665384209 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:27.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.859+0000 7f8d40b73700 1 -- 172.21.15.195:0/665384209 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d30005040 con 0x7f8d3c0f6510 2023-10-12T22:34:27.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.859+0000 7f8d40b73700 1 --2- 172.21.15.195:0/665384209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c106730 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f8d30013510 tx=0x7f8d3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:27.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.859+0000 7f8d397fa700 1 -- 172.21.15.195:0/665384209 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d30009320 con 0x7f8d3c0f6510 2023-10-12T22:34:27.864 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.859+0000 7f8d397fa700 1 -- 172.21.15.195:0/665384209 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d30013f00 con 0x7f8d3c0f6510 2023-10-12T22:34:27.864 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.859+0000 7f8d42dd7700 1 -- 172.21.15.195:0/665384209 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c106e20 con 0x7f8d3c0f6510 2023-10-12T22:34:27.864 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.859+0000 7f8d397fa700 1 -- 172.21.15.195:0/665384209 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d30014d90 con 0x7f8d3c0f6510 2023-10-12T22:34:27.864 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.859+0000 7f8d42dd7700 1 -- 172.21.15.195:0/665384209 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c1072b0 con 0x7f8d3c0f6510 2023-10-12T22:34:27.864 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.860+0000 7f8d397fa700 1 -- 172.21.15.195:0/665384209 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d30014420 con 0x7f8d3c0f6510 2023-10-12T22:34:27.865 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.860+0000 7f8d42dd7700 1 -- 172.21.15.195:0/665384209 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3c045ad0 con 0x7f8d3c0f6510 2023-10-12T22:34:27.865 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.860+0000 7f8d397fa700 1 --2- 172.21.15.195:0/665384209 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d2c02ff60 0x7f8d2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:27.865 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.860+0000 7f8d397fa700 1 -- 172.21.15.195:0/665384209 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d30041000 con 0x7f8d3c0f6510 2023-10-12T22:34:27.865 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.860+0000 7f8d3bfff700 1 --2- 172.21.15.195:0/665384209 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d2c02ff60 0x7f8d2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:27.865 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.861+0000 7f8d3bfff700 1 --2- 172.21.15.195:0/665384209 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d2c02ff60 0x7f8d2c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8d280096a0 tx=0x7f8d28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:27.866 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:27.865+0000 7f8d397fa700 1 -- 172.21.15.195:0/665384209 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d30026bf0 con 0x7f8d3c0f6510 2023-10-12T22:34:28.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:28.070+0000 7f8d42dd7700 1 -- 172.21.15.195:0/665384209 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d3c0fbf60 con 0x7f8d3c0f6510 2023-10-12T22:34:28.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:28.070+0000 7f8d397fa700 1 -- 172.21.15.195:0/665384209 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8d3001db70 con 0x7f8d3c0f6510 2023-10-12T22:34:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:28.071 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:28.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:28.072+0000 7f8d42dd7700 1 -- 172.21.15.195:0/665384209 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d2c02ff60 msgr2=0x7f8d2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:28.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:28.072+0000 7f8d42dd7700 1 --2- 172.21.15.195:0/665384209 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d2c02ff60 0x7f8d2c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8d280096a0 tx=0x7f8d28006b40).stop 2023-10-12T22:34:28.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:28.072+0000 7f8d42dd7700 1 -- 172.21.15.195:0/665384209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d3c0f6510 msgr2=0x7f8d3c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:28.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:28.072+0000 7f8d42dd7700 1 --2- 172.21.15.195:0/665384209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c106730 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f8d30013510 tx=0x7f8d3000b040).stop 2023-10-12T22:34:28.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:28.072+0000 7f8d42dd7700 1 -- 172.21.15.195:0/665384209 shutdown_connections 2023-10-12T22:34:28.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:28.072+0000 7f8d42dd7700 1 --2- 172.21.15.195:0/665384209 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d2c02ff60 0x7f8d2c032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:28.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:28.072+0000 7f8d42dd7700 1 --2- 172.21.15.195:0/665384209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c106730 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:28.074 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:28.072+0000 7f8d42dd7700 1 -- 172.21.15.195:0/665384209 >> 172.21.15.195:0/665384209 conn(0x7f8d3c0f1c70 msgr2=0x7f8d3c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:28.074 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:28.072+0000 7f8d42dd7700 1 -- 172.21.15.195:0/665384209 shutdown_connections 2023-10-12T22:34:28.074 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:28.072+0000 7f8d42dd7700 1 -- 172.21.15.195:0/665384209 wait complete. 2023-10-12T22:34:28.075 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:28.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:28 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:26.825198+0000 mgr.smithi131.efktma (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:28.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:28 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:28.070657+0000 mon.smithi131 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.195:0/665384209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:29.449 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:29.449 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:30.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:30 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:28.825581+0000 mgr.smithi131.efktma (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:30.530 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.528+0000 7f5820513700 1 -- 172.21.15.195:0/2811296979 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58180f5410 msgr2=0x7f58180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:30.530 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.528+0000 7f5820513700 1 --2- 172.21.15.195:0/2811296979 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58180f5410 0x7f58180f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f5814009fb0 tx=0x7f5814009b30).stop 2023-10-12T22:34:30.530 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.528+0000 7f5820513700 1 -- 172.21.15.195:0/2811296979 shutdown_connections 2023-10-12T22:34:30.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.528+0000 7f5820513700 1 --2- 172.21.15.195:0/2811296979 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58180f5410 0x7f58180f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:30.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.528+0000 7f5820513700 1 -- 172.21.15.195:0/2811296979 >> 172.21.15.195:0/2811296979 conn(0x7f58180f0b70 msgr2=0x7f58180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:30.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.528+0000 7f5820513700 1 -- 172.21.15.195:0/2811296979 shutdown_connections 2023-10-12T22:34:30.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.528+0000 7f5820513700 1 -- 172.21.15.195:0/2811296979 wait complete. 2023-10-12T22:34:30.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.529+0000 7f5820513700 1 Processor -- start 2023-10-12T22:34:30.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.529+0000 7f5820513700 1 -- start start 2023-10-12T22:34:30.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.529+0000 7f5820513700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58180f5410 0x7f58180fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:30.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.529+0000 7f5820513700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58180fb820 con 0x7f58180f5410 2023-10-12T22:34:30.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.529+0000 7f581e2af700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58180f5410 0x7f58180fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:30.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.529+0000 7f581e2af700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58180f5410 0x7f58180fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:43618/0 (socket says 172.21.15.195:43618) 2023-10-12T22:34:30.533 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.529+0000 7f581e2af700 1 -- 172.21.15.195:0/197585496 learned_addr learned my addr 172.21.15.195:0/197585496 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:30.533 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.529+0000 7f581e2af700 1 -- 172.21.15.195:0/197585496 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5814005040 con 0x7f58180f5410 2023-10-12T22:34:30.533 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.530+0000 7f581e2af700 1 --2- 172.21.15.195:0/197585496 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58180f5410 0x7f58180fd3a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f5814014000 tx=0x7f581400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:30.533 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.530+0000 7f580effd700 1 -- 172.21.15.195:0/197585496 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58140095f0 con 0x7f58180f5410 2023-10-12T22:34:30.534 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.530+0000 7f5820513700 1 -- 172.21.15.195:0/197585496 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58180fb9c0 con 0x7f58180f5410 2023-10-12T22:34:30.534 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.530+0000 7f580effd700 1 -- 172.21.15.195:0/197585496 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5814007e70 con 0x7f58180f5410 2023-10-12T22:34:30.534 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.530+0000 7f580effd700 1 -- 172.21.15.195:0/197585496 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5814014ea0 con 0x7f58180f5410 2023-10-12T22:34:30.534 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.530+0000 7f5820513700 1 -- 172.21.15.195:0/197585496 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58180fbe50 con 0x7f58180f5410 2023-10-12T22:34:30.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.530+0000 7f580effd700 1 -- 172.21.15.195:0/197585496 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5814025070 con 0x7f58180f5410 2023-10-12T22:34:30.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.530+0000 7f5820513700 1 -- 172.21.15.195:0/197585496 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5818059100 con 0x7f58180f5410 2023-10-12T22:34:30.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.531+0000 7f580effd700 1 --2- 172.21.15.195:0/197585496 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f580402ff60 0x7f5804032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:30.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.531+0000 7f580effd700 1 -- 172.21.15.195:0/197585496 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5814040a00 con 0x7f58180f5410 2023-10-12T22:34:30.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.531+0000 7f581daae700 1 --2- 172.21.15.195:0/197585496 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f580402ff60 0x7f5804032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:30.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.531+0000 7f581daae700 1 --2- 172.21.15.195:0/197585496 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f580402ff60 0x7f5804032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f58080096a0 tx=0x7f5808006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:30.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.535+0000 7f580effd700 1 -- 172.21.15.195:0/197585496 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58140079c0 con 0x7f58180f5410 2023-10-12T22:34:30.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.745+0000 7f5820513700 1 -- 172.21.15.195:0/197585496 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5818045ad0 con 0x7f58180f5410 2023-10-12T22:34:30.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.745+0000 7f580effd700 1 -- 172.21.15.195:0/197585496 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f581400f300 con 0x7f58180f5410 2023-10-12T22:34:30.746 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:30.746 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:30.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.747+0000 7f5820513700 1 -- 172.21.15.195:0/197585496 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f580402ff60 msgr2=0x7f5804032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:30.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.747+0000 7f5820513700 1 --2- 172.21.15.195:0/197585496 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f580402ff60 0x7f5804032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f58080096a0 tx=0x7f5808006b40).stop 2023-10-12T22:34:30.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.747+0000 7f5820513700 1 -- 172.21.15.195:0/197585496 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58180f5410 msgr2=0x7f58180fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:30.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.747+0000 7f5820513700 1 --2- 172.21.15.195:0/197585496 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58180f5410 0x7f58180fd3a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f5814014000 tx=0x7f581400b040).stop 2023-10-12T22:34:30.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.747+0000 7f5820513700 1 -- 172.21.15.195:0/197585496 shutdown_connections 2023-10-12T22:34:30.750 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.747+0000 7f5820513700 1 --2- 172.21.15.195:0/197585496 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f580402ff60 0x7f5804032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:30.750 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.747+0000 7f5820513700 1 --2- 172.21.15.195:0/197585496 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58180f5410 0x7f58180fd3a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:30.751 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.747+0000 7f5820513700 1 -- 172.21.15.195:0/197585496 >> 172.21.15.195:0/197585496 conn(0x7f58180f0b70 msgr2=0x7f581805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:30.751 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.747+0000 7f5820513700 1 -- 172.21.15.195:0/197585496 shutdown_connections 2023-10-12T22:34:30.751 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:30.747+0000 7f5820513700 1 -- 172.21.15.195:0/197585496 wait complete. 2023-10-12T22:34:30.752 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:31.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:31 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:30.745761+0000 mon.smithi131 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.195:0/197585496' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:32.149 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:32.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:32.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:30.825915+0000 mgr.smithi131.efktma (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:33.293 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.293+0000 7f85196bf700 1 -- 172.21.15.195:0/3639213565 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f85140f5410 msgr2=0x7f85140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:33.293 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.293+0000 7f85196bf700 1 --2- 172.21.15.195:0/3639213565 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f85140f5410 0x7f85140f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f8508009fb0 tx=0x7f8508009b30).stop 2023-10-12T22:34:33.293 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.293+0000 7f85196bf700 1 -- 172.21.15.195:0/3639213565 shutdown_connections 2023-10-12T22:34:33.294 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.293+0000 7f85196bf700 1 --2- 172.21.15.195:0/3639213565 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f85140f5410 0x7f85140f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:33.294 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.293+0000 7f85196bf700 1 -- 172.21.15.195:0/3639213565 >> 172.21.15.195:0/3639213565 conn(0x7f85140f0b70 msgr2=0x7f85140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:33.294 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.293+0000 7f85196bf700 1 -- 172.21.15.195:0/3639213565 shutdown_connections 2023-10-12T22:34:33.294 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.293+0000 7f85196bf700 1 -- 172.21.15.195:0/3639213565 wait complete. 2023-10-12T22:34:33.296 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.293+0000 7f85196bf700 1 Processor -- start 2023-10-12T22:34:33.297 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.294+0000 7f85196bf700 1 -- start start 2023-10-12T22:34:33.297 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.294+0000 7f85196bf700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f85140f5410 0x7f8514109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:33.297 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.294+0000 7f85196bf700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8514060240 con 0x7f85140f5410 2023-10-12T22:34:33.298 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.294+0000 7f8512ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f85140f5410 0x7f8514109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:33.298 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.294+0000 7f8512ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f85140f5410 0x7f8514109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:43640/0 (socket says 172.21.15.195:43640) 2023-10-12T22:34:33.298 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.294+0000 7f8512ffd700 1 -- 172.21.15.195:0/3736376712 learned_addr learned my addr 172.21.15.195:0/3736376712 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:33.298 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.294+0000 7f8512ffd700 1 -- 172.21.15.195:0/3736376712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8508005040 con 0x7f85140f5410 2023-10-12T22:34:33.298 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.295+0000 7f8512ffd700 1 --2- 172.21.15.195:0/3736376712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f85140f5410 0x7f8514109400 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8508013510 tx=0x7f850800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:33.299 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.295+0000 7f84fb7fe700 1 -- 172.21.15.195:0/3736376712 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8508009320 con 0x7f85140f5410 2023-10-12T22:34:33.299 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.295+0000 7f85196bf700 1 -- 172.21.15.195:0/3736376712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8514060440 con 0x7f85140f5410 2023-10-12T22:34:33.299 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.295+0000 7f84fb7fe700 1 -- 172.21.15.195:0/3736376712 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8508013f00 con 0x7f85140f5410 2023-10-12T22:34:33.299 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.295+0000 7f84fb7fe700 1 -- 172.21.15.195:0/3736376712 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8508014d90 con 0x7f85140f5410 2023-10-12T22:34:33.299 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.295+0000 7f85196bf700 1 -- 172.21.15.195:0/3736376712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8514109ab0 con 0x7f85140f5410 2023-10-12T22:34:33.299 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.296+0000 7f84fb7fe700 1 -- 172.21.15.195:0/3736376712 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8508014420 con 0x7f85140f5410 2023-10-12T22:34:33.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.296+0000 7f85196bf700 1 -- 172.21.15.195:0/3736376712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8514045ad0 con 0x7f85140f5410 2023-10-12T22:34:33.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.296+0000 7f84fb7fe700 1 --2- 172.21.15.195:0/3736376712 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f84fc02ff60 0x7f84fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:33.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.296+0000 7f84fb7fe700 1 -- 172.21.15.195:0/3736376712 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85080410c0 con 0x7f85140f5410 2023-10-12T22:34:33.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.296+0000 7f85127fc700 1 --2- 172.21.15.195:0/3736376712 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f84fc02ff60 0x7f84fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:33.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.296+0000 7f85127fc700 1 --2- 172.21.15.195:0/3736376712 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f84fc02ff60 0x7f84fc032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f85040096a0 tx=0x7f8504006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:33.301 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.301+0000 7f84fb7fe700 1 -- 172.21.15.195:0/3736376712 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85080095f0 con 0x7f85140f5410 2023-10-12T22:34:33.514 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.513+0000 7f85196bf700 1 -- 172.21.15.195:0/3736376712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85140fae60 con 0x7f85140f5410 2023-10-12T22:34:33.514 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.514+0000 7f84fb7fe700 1 -- 172.21.15.195:0/3736376712 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f850801db70 con 0x7f85140f5410 2023-10-12T22:34:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:33.515 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:33.516 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.515+0000 7f85196bf700 1 -- 172.21.15.195:0/3736376712 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f84fc02ff60 msgr2=0x7f84fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:33.516 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.515+0000 7f85196bf700 1 --2- 172.21.15.195:0/3736376712 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f84fc02ff60 0x7f84fc032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f85040096a0 tx=0x7f8504006b40).stop 2023-10-12T22:34:33.516 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.515+0000 7f85196bf700 1 -- 172.21.15.195:0/3736376712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f85140f5410 msgr2=0x7f8514109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:33.516 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.515+0000 7f85196bf700 1 --2- 172.21.15.195:0/3736376712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f85140f5410 0x7f8514109400 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8508013510 tx=0x7f850800b040).stop 2023-10-12T22:34:33.517 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.516+0000 7f85196bf700 1 -- 172.21.15.195:0/3736376712 shutdown_connections 2023-10-12T22:34:33.517 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.516+0000 7f85196bf700 1 --2- 172.21.15.195:0/3736376712 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f84fc02ff60 0x7f84fc032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:33.517 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.516+0000 7f85196bf700 1 --2- 172.21.15.195:0/3736376712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f85140f5410 0x7f8514109400 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:33.517 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.516+0000 7f85196bf700 1 -- 172.21.15.195:0/3736376712 >> 172.21.15.195:0/3736376712 conn(0x7f85140f0b70 msgr2=0x7f85140f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:33.517 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.516+0000 7f85196bf700 1 -- 172.21.15.195:0/3736376712 shutdown_connections 2023-10-12T22:34:33.517 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:33.516+0000 7f85196bf700 1 -- 172.21.15.195:0/3736376712 wait complete. 2023-10-12T22:34:33.518 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:34.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:32.826262+0000 mgr.smithi131.efktma (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:34.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:33.514400+0000 mon.smithi131 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.195:0/3736376712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:34.911 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:34.912 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:36.113 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.112+0000 7f7993aff700 1 -- 172.21.15.195:0/2438877658 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f798c060280 msgr2=0x7f798c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:36.113 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.112+0000 7f7993aff700 1 --2- 172.21.15.195:0/2438877658 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f798c060280 0x7f798c0f90f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f7988009fb0 tx=0x7f7988009b30).stop 2023-10-12T22:34:36.113 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.112+0000 7f7993aff700 1 -- 172.21.15.195:0/2438877658 shutdown_connections 2023-10-12T22:34:36.114 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.112+0000 7f7993aff700 1 --2- 172.21.15.195:0/2438877658 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f798c060280 0x7f798c0f90f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:36.114 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.112+0000 7f7993aff700 1 -- 172.21.15.195:0/2438877658 >> 172.21.15.195:0/2438877658 conn(0x7f798c0f0b70 msgr2=0x7f798c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:36.114 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.112+0000 7f7993aff700 1 -- 172.21.15.195:0/2438877658 shutdown_connections 2023-10-12T22:34:36.114 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.112+0000 7f7993aff700 1 -- 172.21.15.195:0/2438877658 wait complete. 2023-10-12T22:34:36.114 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.112+0000 7f7993aff700 1 Processor -- start 2023-10-12T22:34:36.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.112+0000 7f7993aff700 1 -- start start 2023-10-12T22:34:36.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.113+0000 7f7993aff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f798c060280 0x7f798c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:36.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.113+0000 7f7993aff700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0f4ff0 con 0x7f798c060280 2023-10-12T22:34:36.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.113+0000 7f799189b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f798c060280 0x7f798c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:36.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.113+0000 7f799189b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f798c060280 0x7f798c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:43664/0 (socket says 172.21.15.195:43664) 2023-10-12T22:34:36.116 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.113+0000 7f799189b700 1 -- 172.21.15.195:0/2238360495 learned_addr learned my addr 172.21.15.195:0/2238360495 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:36.117 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.113+0000 7f799189b700 1 -- 172.21.15.195:0/2238360495 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7988005040 con 0x7f798c060280 2023-10-12T22:34:36.117 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.113+0000 7f799189b700 1 --2- 172.21.15.195:0/2238360495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f798c060280 0x7f798c0f6b70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f7988013510 tx=0x7f798800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:36.118 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.115+0000 7f79827fc700 1 -- 172.21.15.195:0/2238360495 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7988009320 con 0x7f798c060280 2023-10-12T22:34:36.118 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.115+0000 7f79827fc700 1 -- 172.21.15.195:0/2238360495 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7988013f00 con 0x7f798c060280 2023-10-12T22:34:36.118 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.115+0000 7f7993aff700 1 -- 172.21.15.195:0/2238360495 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798c0f5190 con 0x7f798c060280 2023-10-12T22:34:36.118 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.115+0000 7f79827fc700 1 -- 172.21.15.195:0/2238360495 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7988014dc0 con 0x7f798c060280 2023-10-12T22:34:36.119 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.115+0000 7f7993aff700 1 -- 172.21.15.195:0/2238360495 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798c0f5620 con 0x7f798c060280 2023-10-12T22:34:36.119 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.116+0000 7f7993aff700 1 -- 172.21.15.195:0/2238360495 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798c059100 con 0x7f798c060280 2023-10-12T22:34:36.119 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.116+0000 7f79827fc700 1 -- 172.21.15.195:0/2238360495 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79880095f0 con 0x7f798c060280 2023-10-12T22:34:36.119 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.116+0000 7f79827fc700 1 --2- 172.21.15.195:0/2238360495 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f797802ffb0 0x7f7978032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:36.119 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.116+0000 7f79827fc700 1 -- 172.21.15.195:0/2238360495 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f79880410b0 con 0x7f798c060280 2023-10-12T22:34:36.119 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.116+0000 7f799109a700 1 --2- 172.21.15.195:0/2238360495 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f797802ffb0 0x7f7978032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:36.120 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.117+0000 7f799109a700 1 --2- 172.21.15.195:0/2238360495 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f797802ffb0 0x7f7978032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f797c0096a0 tx=0x7f797c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:36.121 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.121+0000 7f79827fc700 1 -- 172.21.15.195:0/2238360495 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f798806d2b0 con 0x7f798c060280 2023-10-12T22:34:36.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.328+0000 7f7993aff700 1 -- 172.21.15.195:0/2238360495 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f798c0fbf50 con 0x7f798c060280 2023-10-12T22:34:36.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.328+0000 7f79827fc700 1 -- 172.21.15.195:0/2238360495 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7988014420 con 0x7f798c060280 2023-10-12T22:34:36.329 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:36.329 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:36.330 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.330+0000 7f7993aff700 1 -- 172.21.15.195:0/2238360495 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f797802ffb0 msgr2=0x7f7978032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:36.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.330+0000 7f7993aff700 1 --2- 172.21.15.195:0/2238360495 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f797802ffb0 0x7f7978032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f797c0096a0 tx=0x7f797c006b40).stop 2023-10-12T22:34:36.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.330+0000 7f7993aff700 1 -- 172.21.15.195:0/2238360495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f798c060280 msgr2=0x7f798c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:36.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.330+0000 7f7993aff700 1 --2- 172.21.15.195:0/2238360495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f798c060280 0x7f798c0f6b70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f7988013510 tx=0x7f798800b040).stop 2023-10-12T22:34:36.331 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.330+0000 7f7993aff700 1 -- 172.21.15.195:0/2238360495 shutdown_connections 2023-10-12T22:34:36.332 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.330+0000 7f7993aff700 1 --2- 172.21.15.195:0/2238360495 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f797802ffb0 0x7f7978032470 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:36.332 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.330+0000 7f7993aff700 1 --2- 172.21.15.195:0/2238360495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f798c060280 0x7f798c0f6b70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:36.332 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.330+0000 7f7993aff700 1 -- 172.21.15.195:0/2238360495 >> 172.21.15.195:0/2238360495 conn(0x7f798c0f0b70 msgr2=0x7f798c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:36.332 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.330+0000 7f7993aff700 1 -- 172.21.15.195:0/2238360495 shutdown_connections 2023-10-12T22:34:36.332 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:36.330+0000 7f7993aff700 1 -- 172.21.15.195:0/2238360495 wait complete. 2023-10-12T22:34:36.333 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:36.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:36 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:34.826547+0000 mgr.smithi131.efktma (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:37.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:37 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:36.328738+0000 mon.smithi131 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.195:0/2238360495' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:37.708 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:37.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:38.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:38 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:36.826893+0000 mgr.smithi131.efktma (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:38.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.869+0000 7ff0e2b73700 1 -- 172.21.15.195:0/2370423618 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff0dc0f5410 msgr2=0x7ff0dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:38.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.869+0000 7ff0e2b73700 1 --2- 172.21.15.195:0/2370423618 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff0dc0f5410 0x7ff0dc0f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7ff0d0009fb0 tx=0x7ff0d0009b30).stop 2023-10-12T22:34:38.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.870+0000 7ff0e2b73700 1 -- 172.21.15.195:0/2370423618 shutdown_connections 2023-10-12T22:34:38.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.870+0000 7ff0e2b73700 1 --2- 172.21.15.195:0/2370423618 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff0dc0f5410 0x7ff0dc0f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:38.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.870+0000 7ff0e2b73700 1 -- 172.21.15.195:0/2370423618 >> 172.21.15.195:0/2370423618 conn(0x7ff0dc0f0b70 msgr2=0x7ff0dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:38.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.870+0000 7ff0e2b73700 1 -- 172.21.15.195:0/2370423618 shutdown_connections 2023-10-12T22:34:38.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.870+0000 7ff0e2b73700 1 -- 172.21.15.195:0/2370423618 wait complete. 2023-10-12T22:34:38.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.870+0000 7ff0e2b73700 1 Processor -- start 2023-10-12T22:34:38.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.870+0000 7ff0e2b73700 1 -- start start 2023-10-12T22:34:38.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.870+0000 7ff0e2b73700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff0dc0f5410 0x7ff0dc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:38.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.870+0000 7ff0e2b73700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc060220 con 0x7ff0dc0f5410 2023-10-12T22:34:38.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.871+0000 7ff0e090f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff0dc0f5410 0x7ff0dc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:38.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.871+0000 7ff0e090f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff0dc0f5410 0x7ff0dc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:40810/0 (socket says 172.21.15.195:40810) 2023-10-12T22:34:38.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.871+0000 7ff0e090f700 1 -- 172.21.15.195:0/180333718 learned_addr learned my addr 172.21.15.195:0/180333718 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:38.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.871+0000 7ff0e090f700 1 -- 172.21.15.195:0/180333718 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d0005040 con 0x7ff0dc0f5410 2023-10-12T22:34:38.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.871+0000 7ff0e090f700 1 --2- 172.21.15.195:0/180333718 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff0dc0f5410 0x7ff0dc109450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7ff0d0013510 tx=0x7ff0d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:38.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.871+0000 7ff0d97fa700 1 -- 172.21.15.195:0/180333718 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0d0009320 con 0x7ff0dc0f5410 2023-10-12T22:34:38.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.871+0000 7ff0e2b73700 1 -- 172.21.15.195:0/180333718 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0dc060420 con 0x7ff0dc0f5410 2023-10-12T22:34:38.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.871+0000 7ff0d97fa700 1 -- 172.21.15.195:0/180333718 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0d0013f00 con 0x7ff0dc0f5410 2023-10-12T22:34:38.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.871+0000 7ff0d97fa700 1 -- 172.21.15.195:0/180333718 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0d0014d90 con 0x7ff0dc0f5410 2023-10-12T22:34:38.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.871+0000 7ff0e2b73700 1 -- 172.21.15.195:0/180333718 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0dc109b00 con 0x7ff0dc0f5410 2023-10-12T22:34:38.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.872+0000 7ff0d97fa700 1 -- 172.21.15.195:0/180333718 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0d00095f0 con 0x7ff0dc0f5410 2023-10-12T22:34:38.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.872+0000 7ff0e2b73700 1 -- 172.21.15.195:0/180333718 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0dc059100 con 0x7ff0dc0f5410 2023-10-12T22:34:38.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.872+0000 7ff0d97fa700 1 --2- 172.21.15.195:0/180333718 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff0cc02ff60 0x7ff0cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:38.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.872+0000 7ff0d97fa700 1 -- 172.21.15.195:0/180333718 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff0d0041000 con 0x7ff0dc0f5410 2023-10-12T22:34:38.878 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.872+0000 7ff0dbfff700 1 --2- 172.21.15.195:0/180333718 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff0cc02ff60 0x7ff0cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:38.878 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.873+0000 7ff0dbfff700 1 --2- 172.21.15.195:0/180333718 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff0cc02ff60 0x7ff0cc032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff0c80096a0 tx=0x7ff0c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:38.878 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:38.877+0000 7ff0d97fa700 1 -- 172.21.15.195:0/180333718 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0d0014420 con 0x7ff0dc0f5410 2023-10-12T22:34:39.083 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:39.083+0000 7ff0e2b73700 1 -- 172.21.15.195:0/180333718 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0dc045ad0 con 0x7ff0dc0f5410 2023-10-12T22:34:39.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:39.084+0000 7ff0d97fa700 1 -- 172.21.15.195:0/180333718 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff0d0010230 con 0x7ff0dc0f5410 2023-10-12T22:34:39.084 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:39.085 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:39.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:39.086+0000 7ff0e2b73700 1 -- 172.21.15.195:0/180333718 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff0cc02ff60 msgr2=0x7ff0cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:39.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:39.086+0000 7ff0e2b73700 1 --2- 172.21.15.195:0/180333718 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff0cc02ff60 0x7ff0cc032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff0c80096a0 tx=0x7ff0c8006b40).stop 2023-10-12T22:34:39.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:39.086+0000 7ff0e2b73700 1 -- 172.21.15.195:0/180333718 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff0dc0f5410 msgr2=0x7ff0dc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:39.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:39.086+0000 7ff0e2b73700 1 --2- 172.21.15.195:0/180333718 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff0dc0f5410 0x7ff0dc109450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7ff0d0013510 tx=0x7ff0d000b040).stop 2023-10-12T22:34:39.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:39.086+0000 7ff0e2b73700 1 -- 172.21.15.195:0/180333718 shutdown_connections 2023-10-12T22:34:39.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:39.086+0000 7ff0e2b73700 1 --2- 172.21.15.195:0/180333718 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff0cc02ff60 0x7ff0cc032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:39.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:39.086+0000 7ff0e2b73700 1 --2- 172.21.15.195:0/180333718 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff0dc0f5410 0x7ff0dc109450 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:39.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:39.086+0000 7ff0e2b73700 1 -- 172.21.15.195:0/180333718 >> 172.21.15.195:0/180333718 conn(0x7ff0dc0f0b70 msgr2=0x7ff0dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:39.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:39.086+0000 7ff0e2b73700 1 -- 172.21.15.195:0/180333718 shutdown_connections 2023-10-12T22:34:39.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:39.086+0000 7ff0e2b73700 1 -- 172.21.15.195:0/180333718 wait complete. 2023-10-12T22:34:39.089 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:39.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:39 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:39.084169+0000 mon.smithi131 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.195:0/180333718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:40.458 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:40.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:40.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:40 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:38.827277+0000 mgr.smithi131.efktma (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:41.555 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.551+0000 7f1d9da37700 1 -- 172.21.15.195:0/3229357245 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1d980f47c0 msgr2=0x7f1d980f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:41.555 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.551+0000 7f1d9da37700 1 --2- 172.21.15.195:0/3229357245 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1d980f47c0 0x7f1d980f4be0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f1d8c009fb0 tx=0x7f1d8c009b30).stop 2023-10-12T22:34:41.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.551+0000 7f1d9da37700 1 -- 172.21.15.195:0/3229357245 shutdown_connections 2023-10-12T22:34:41.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.551+0000 7f1d9da37700 1 --2- 172.21.15.195:0/3229357245 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1d980f47c0 0x7f1d980f4be0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:41.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.551+0000 7f1d9da37700 1 -- 172.21.15.195:0/3229357245 >> 172.21.15.195:0/3229357245 conn(0x7f1d980f02e0 msgr2=0x7f1d980f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:41.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.551+0000 7f1d9da37700 1 -- 172.21.15.195:0/3229357245 shutdown_connections 2023-10-12T22:34:41.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.551+0000 7f1d9da37700 1 -- 172.21.15.195:0/3229357245 wait complete. 2023-10-12T22:34:41.557 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.552+0000 7f1d9da37700 1 Processor -- start 2023-10-12T22:34:41.557 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.552+0000 7f1d9da37700 1 -- start start 2023-10-12T22:34:41.557 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.552+0000 7f1d9da37700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1d980f47c0 0x7f1d98106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:41.557 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.552+0000 7f1d9da37700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d98106bf0 con 0x7f1d980f47c0 2023-10-12T22:34:41.557 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.552+0000 7f1d96ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1d980f47c0 0x7f1d98106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:41.557 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.552+0000 7f1d96ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1d980f47c0 0x7f1d98106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:40832/0 (socket says 172.21.15.195:40832) 2023-10-12T22:34:41.558 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.552+0000 7f1d96ffd700 1 -- 172.21.15.195:0/1076758400 learned_addr learned my addr 172.21.15.195:0/1076758400 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:41.558 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.553+0000 7f1d96ffd700 1 -- 172.21.15.195:0/1076758400 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c005040 con 0x7f1d980f47c0 2023-10-12T22:34:41.558 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.553+0000 7f1d96ffd700 1 --2- 172.21.15.195:0/1076758400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1d980f47c0 0x7f1d98106700 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f1d8c013510 tx=0x7f1d8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:41.558 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.553+0000 7f1d7f7fe700 1 -- 172.21.15.195:0/1076758400 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d8c009320 con 0x7f1d980f47c0 2023-10-12T22:34:41.558 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.553+0000 7f1d9da37700 1 -- 172.21.15.195:0/1076758400 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d98106df0 con 0x7f1d980f47c0 2023-10-12T22:34:41.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.553+0000 7f1d7f7fe700 1 -- 172.21.15.195:0/1076758400 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d8c013f00 con 0x7f1d980f47c0 2023-10-12T22:34:41.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.553+0000 7f1d7f7fe700 1 -- 172.21.15.195:0/1076758400 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d8c014d90 con 0x7f1d980f47c0 2023-10-12T22:34:41.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.553+0000 7f1d9da37700 1 -- 172.21.15.195:0/1076758400 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d98107280 con 0x7f1d980f47c0 2023-10-12T22:34:41.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.554+0000 7f1d7f7fe700 1 -- 172.21.15.195:0/1076758400 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d8c0263a0 con 0x7f1d980f47c0 2023-10-12T22:34:41.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.554+0000 7f1d9da37700 1 -- 172.21.15.195:0/1076758400 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d98045ad0 con 0x7f1d980f47c0 2023-10-12T22:34:41.560 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.554+0000 7f1d7f7fe700 1 --2- 172.21.15.195:0/1076758400 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1d8002ff60 0x7f1d80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:41.560 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.554+0000 7f1d7f7fe700 1 -- 172.21.15.195:0/1076758400 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d8c0413c0 con 0x7f1d980f47c0 2023-10-12T22:34:41.561 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.554+0000 7f1d967fc700 1 --2- 172.21.15.195:0/1076758400 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1d8002ff60 0x7f1d80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:41.561 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.555+0000 7f1d967fc700 1 --2- 172.21.15.195:0/1076758400 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1d8002ff60 0x7f1d80032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1d880096a0 tx=0x7f1d88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:41.561 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.559+0000 7f1d7f7fe700 1 -- 172.21.15.195:0/1076758400 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d8c01cb10 con 0x7f1d980f47c0 2023-10-12T22:34:41.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.765+0000 7f1d9da37700 1 -- 172.21.15.195:0/1076758400 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d98108130 con 0x7f1d980f47c0 2023-10-12T22:34:41.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.766+0000 7f1d7f7fe700 1 -- 172.21.15.195:0/1076758400 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1d8c01cb10 con 0x7f1d980f47c0 2023-10-12T22:34:41.768 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:41.768 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:41.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.768+0000 7f1d9da37700 1 -- 172.21.15.195:0/1076758400 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1d8002ff60 msgr2=0x7f1d80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:41.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.768+0000 7f1d9da37700 1 --2- 172.21.15.195:0/1076758400 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1d8002ff60 0x7f1d80032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1d880096a0 tx=0x7f1d88006b40).stop 2023-10-12T22:34:41.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.768+0000 7f1d9da37700 1 -- 172.21.15.195:0/1076758400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1d980f47c0 msgr2=0x7f1d98106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:41.770 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.768+0000 7f1d9da37700 1 --2- 172.21.15.195:0/1076758400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1d980f47c0 0x7f1d98106700 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f1d8c013510 tx=0x7f1d8c00b040).stop 2023-10-12T22:34:41.770 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.768+0000 7f1d9da37700 1 -- 172.21.15.195:0/1076758400 shutdown_connections 2023-10-12T22:34:41.770 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.768+0000 7f1d9da37700 1 --2- 172.21.15.195:0/1076758400 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1d8002ff60 0x7f1d80032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:41.770 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.768+0000 7f1d9da37700 1 --2- 172.21.15.195:0/1076758400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1d980f47c0 0x7f1d98106700 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:41.770 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.768+0000 7f1d9da37700 1 -- 172.21.15.195:0/1076758400 >> 172.21.15.195:0/1076758400 conn(0x7f1d980f02e0 msgr2=0x7f1d980f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:41.771 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.768+0000 7f1d9da37700 1 -- 172.21.15.195:0/1076758400 shutdown_connections 2023-10-12T22:34:41.771 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:41.768+0000 7f1d9da37700 1 -- 172.21.15.195:0/1076758400 wait complete. 2023-10-12T22:34:41.771 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:42.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:42 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:40.827517+0000 mgr.smithi131.efktma (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:42.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:42 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:41.767444+0000 mon.smithi131 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.195:0/1076758400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:43.158 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:43.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:44.311 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.305+0000 7f03ad9d5700 1 -- 172.21.15.195:0/1707462000 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03a80f62a0 msgr2=0x7f03a80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:44.312 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.305+0000 7f03ad9d5700 1 --2- 172.21.15.195:0/1707462000 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03a80f62a0 0x7f03a80f66c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f039c009fb0 tx=0x7f039c009b30).stop 2023-10-12T22:34:44.312 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.305+0000 7f03ad9d5700 1 -- 172.21.15.195:0/1707462000 shutdown_connections 2023-10-12T22:34:44.312 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.305+0000 7f03ad9d5700 1 --2- 172.21.15.195:0/1707462000 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03a80f62a0 0x7f03a80f66c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:44.312 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.305+0000 7f03ad9d5700 1 -- 172.21.15.195:0/1707462000 >> 172.21.15.195:0/1707462000 conn(0x7f03a80f1aa0 msgr2=0x7f03a80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:44.312 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.305+0000 7f03ad9d5700 1 -- 172.21.15.195:0/1707462000 shutdown_connections 2023-10-12T22:34:44.313 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.305+0000 7f03ad9d5700 1 -- 172.21.15.195:0/1707462000 wait complete. 2023-10-12T22:34:44.313 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.306+0000 7f03ad9d5700 1 Processor -- start 2023-10-12T22:34:44.313 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.306+0000 7f03ad9d5700 1 -- start start 2023-10-12T22:34:44.313 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.306+0000 7f03ad9d5700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03a80f62a0 0x7f03a81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:44.313 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.306+0000 7f03ad9d5700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a81069a0 con 0x7f03a80f62a0 2023-10-12T22:34:44.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.306+0000 7f03ac9d3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03a80f62a0 0x7f03a81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:44.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.306+0000 7f03ac9d3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03a80f62a0 0x7f03a81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:40854/0 (socket says 172.21.15.195:40854) 2023-10-12T22:34:44.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.306+0000 7f03ac9d3700 1 -- 172.21.15.195:0/2961293754 learned_addr learned my addr 172.21.15.195:0/2961293754 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:44.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.307+0000 7f03ac9d3700 1 -- 172.21.15.195:0/2961293754 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c005040 con 0x7f03a80f62a0 2023-10-12T22:34:44.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.307+0000 7f03ac9d3700 1 --2- 172.21.15.195:0/2961293754 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03a80f62a0 0x7f03a81064b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f039c013510 tx=0x7f039c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:44.315 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.307+0000 7f03a57fa700 1 -- 172.21.15.195:0/2961293754 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f039c009320 con 0x7f03a80f62a0 2023-10-12T22:34:44.315 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.307+0000 7f03a57fa700 1 -- 172.21.15.195:0/2961293754 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f039c013f00 con 0x7f03a80f62a0 2023-10-12T22:34:44.315 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.307+0000 7f03a57fa700 1 -- 172.21.15.195:0/2961293754 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f039c014dc0 con 0x7f03a80f62a0 2023-10-12T22:34:44.315 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.307+0000 7f03ad9d5700 1 -- 172.21.15.195:0/2961293754 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a8106ba0 con 0x7f03a80f62a0 2023-10-12T22:34:44.315 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.307+0000 7f03ad9d5700 1 -- 172.21.15.195:0/2961293754 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a8107030 con 0x7f03a80f62a0 2023-10-12T22:34:44.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.308+0000 7f03a57fa700 1 -- 172.21.15.195:0/2961293754 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f039c014420 con 0x7f03a80f62a0 2023-10-12T22:34:44.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.308+0000 7f03ad9d5700 1 -- 172.21.15.195:0/2961293754 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a8045ad0 con 0x7f03a80f62a0 2023-10-12T22:34:44.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.308+0000 7f03a57fa700 1 --2- 172.21.15.195:0/2961293754 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f039802ff60 0x7f0398032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:44.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.308+0000 7f03a57fa700 1 -- 172.21.15.195:0/2961293754 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f039c0410b0 con 0x7f03a80f62a0 2023-10-12T22:34:44.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.308+0000 7f03a7fff700 1 --2- 172.21.15.195:0/2961293754 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f039802ff60 0x7f0398032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:44.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.309+0000 7f03a7fff700 1 --2- 172.21.15.195:0/2961293754 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f039802ff60 0x7f0398032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f03940096a0 tx=0x7f0394006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:44.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.313+0000 7f03a57fa700 1 -- 172.21.15.195:0/2961293754 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f039c0095f0 con 0x7f03a80f62a0 2023-10-12T22:34:44.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:44 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:42.827851+0000 mgr.smithi131.efktma (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:44.521 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.520+0000 7f03ad9d5700 1 -- 172.21.15.195:0/2961293754 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03a8107ee0 con 0x7f03a80f62a0 2023-10-12T22:34:44.522 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.520+0000 7f03a57fa700 1 -- 172.21.15.195:0/2961293754 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f039c01cb10 con 0x7f03a80f62a0 2023-10-12T22:34:44.522 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:44.522 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:44.523 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.522+0000 7f03ad9d5700 1 -- 172.21.15.195:0/2961293754 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f039802ff60 msgr2=0x7f0398032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:44.524 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.522+0000 7f03ad9d5700 1 --2- 172.21.15.195:0/2961293754 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f039802ff60 0x7f0398032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f03940096a0 tx=0x7f0394006b40).stop 2023-10-12T22:34:44.524 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.522+0000 7f03ad9d5700 1 -- 172.21.15.195:0/2961293754 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03a80f62a0 msgr2=0x7f03a81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:44.524 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.522+0000 7f03ad9d5700 1 --2- 172.21.15.195:0/2961293754 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03a80f62a0 0x7f03a81064b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f039c013510 tx=0x7f039c00b040).stop 2023-10-12T22:34:44.524 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.522+0000 7f03ad9d5700 1 -- 172.21.15.195:0/2961293754 shutdown_connections 2023-10-12T22:34:44.524 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.522+0000 7f03ad9d5700 1 --2- 172.21.15.195:0/2961293754 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f039802ff60 0x7f0398032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:44.525 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.522+0000 7f03ad9d5700 1 --2- 172.21.15.195:0/2961293754 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03a80f62a0 0x7f03a81064b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:44.525 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.522+0000 7f03ad9d5700 1 -- 172.21.15.195:0/2961293754 >> 172.21.15.195:0/2961293754 conn(0x7f03a80f1aa0 msgr2=0x7f03a80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:44.525 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.522+0000 7f03ad9d5700 1 -- 172.21.15.195:0/2961293754 shutdown_connections 2023-10-12T22:34:44.525 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:44.522+0000 7f03ad9d5700 1 -- 172.21.15.195:0/2961293754 wait complete. 2023-10-12T22:34:44.526 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:45.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:45 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:44.521813+0000 mon.smithi131 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.195:0/2961293754' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:45.917 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:45.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:46.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:46 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:44.828217+0000 mgr.smithi131.efktma (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:47.068 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.066+0000 7f522914c700 1 -- 172.21.15.195:0/3007516677 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f52240f4920 msgr2=0x7f52240f4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:47.069 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.066+0000 7f522914c700 1 --2- 172.21.15.195:0/3007516677 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f52240f4920 0x7f52240f4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f5214009fb0 tx=0x7f5214009b30).stop 2023-10-12T22:34:47.069 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.066+0000 7f522914c700 1 -- 172.21.15.195:0/3007516677 shutdown_connections 2023-10-12T22:34:47.069 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.066+0000 7f522914c700 1 --2- 172.21.15.195:0/3007516677 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f52240f4920 0x7f52240f4d40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:47.069 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.066+0000 7f522914c700 1 -- 172.21.15.195:0/3007516677 >> 172.21.15.195:0/3007516677 conn(0x7f52240f0120 msgr2=0x7f52240f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:47.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.066+0000 7f522914c700 1 -- 172.21.15.195:0/3007516677 shutdown_connections 2023-10-12T22:34:47.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.066+0000 7f522914c700 1 -- 172.21.15.195:0/3007516677 wait complete. 2023-10-12T22:34:47.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.066+0000 7f522914c700 1 Processor -- start 2023-10-12T22:34:47.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.067+0000 7f522914c700 1 -- start start 2023-10-12T22:34:47.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.067+0000 7f522914c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f52240f4920 0x7f52241053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:47.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.067+0000 7f522914c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52241058a0 con 0x7f52240f4920 2023-10-12T22:34:47.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.067+0000 7f5223fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f52240f4920 0x7f52241053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:47.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.067+0000 7f5223fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f52240f4920 0x7f52241053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:40862/0 (socket says 172.21.15.195:40862) 2023-10-12T22:34:47.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.067+0000 7f5223fff700 1 -- 172.21.15.195:0/376813505 learned_addr learned my addr 172.21.15.195:0/376813505 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:47.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.067+0000 7f5223fff700 1 -- 172.21.15.195:0/376813505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5214005040 con 0x7f52240f4920 2023-10-12T22:34:47.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.067+0000 7f5223fff700 1 --2- 172.21.15.195:0/376813505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f52240f4920 0x7f52241053b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f5214013510 tx=0x7f521400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:47.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.068+0000 7f5220ff9700 1 -- 172.21.15.195:0/376813505 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5214009320 con 0x7f52240f4920 2023-10-12T22:34:47.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.068+0000 7f522914c700 1 -- 172.21.15.195:0/376813505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5224105aa0 con 0x7f52240f4920 2023-10-12T22:34:47.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.068+0000 7f5220ff9700 1 -- 172.21.15.195:0/376813505 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5214013f00 con 0x7f52240f4920 2023-10-12T22:34:47.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.068+0000 7f5220ff9700 1 -- 172.21.15.195:0/376813505 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5214014dc0 con 0x7f52240f4920 2023-10-12T22:34:47.074 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.068+0000 7f522914c700 1 -- 172.21.15.195:0/376813505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5224105f30 con 0x7f52240f4920 2023-10-12T22:34:47.074 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.068+0000 7f5220ff9700 1 -- 172.21.15.195:0/376813505 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f521401d3d0 con 0x7f52240f4920 2023-10-12T22:34:47.074 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.069+0000 7f5220ff9700 1 --2- 172.21.15.195:0/376813505 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f520c02ff60 0x7f520c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:47.074 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.069+0000 7f5220ff9700 1 -- 172.21.15.195:0/376813505 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5214041110 con 0x7f52240f4920 2023-10-12T22:34:47.075 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.069+0000 7f522914c700 1 -- 172.21.15.195:0/376813505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5224045ad0 con 0x7f52240f4920 2023-10-12T22:34:47.075 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.069+0000 7f52237fe700 1 --2- 172.21.15.195:0/376813505 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f520c02ff60 0x7f520c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:47.075 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.069+0000 7f52237fe700 1 --2- 172.21.15.195:0/376813505 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f520c02ff60 0x7f520c032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f52180096a0 tx=0x7f5218006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:47.075 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.073+0000 7f5220ff9700 1 -- 172.21.15.195:0/376813505 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5214012920 con 0x7f52240f4920 2023-10-12T22:34:47.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.282+0000 7f522914c700 1 -- 172.21.15.195:0/376813505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52240ff380 con 0x7f52240f4920 2023-10-12T22:34:47.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.283+0000 7f5220ff9700 1 -- 172.21.15.195:0/376813505 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f521402b020 con 0x7f52240f4920 2023-10-12T22:34:47.285 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:47.285 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:47.286 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.284+0000 7f522914c700 1 -- 172.21.15.195:0/376813505 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f520c02ff60 msgr2=0x7f520c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:47.286 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.284+0000 7f522914c700 1 --2- 172.21.15.195:0/376813505 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f520c02ff60 0x7f520c032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f52180096a0 tx=0x7f5218006b40).stop 2023-10-12T22:34:47.286 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.284+0000 7f522914c700 1 -- 172.21.15.195:0/376813505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f52240f4920 msgr2=0x7f52241053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:47.286 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.284+0000 7f522914c700 1 --2- 172.21.15.195:0/376813505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f52240f4920 0x7f52241053b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f5214013510 tx=0x7f521400b040).stop 2023-10-12T22:34:47.287 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.284+0000 7f522914c700 1 -- 172.21.15.195:0/376813505 shutdown_connections 2023-10-12T22:34:47.287 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.285+0000 7f522914c700 1 --2- 172.21.15.195:0/376813505 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f520c02ff60 0x7f520c032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:47.287 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.285+0000 7f522914c700 1 --2- 172.21.15.195:0/376813505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f52240f4920 0x7f52241053b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:47.287 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.285+0000 7f522914c700 1 -- 172.21.15.195:0/376813505 >> 172.21.15.195:0/376813505 conn(0x7f52240f0120 msgr2=0x7f52240f2380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:47.287 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.285+0000 7f522914c700 1 -- 172.21.15.195:0/376813505 shutdown_connections 2023-10-12T22:34:47.288 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:47.285+0000 7f522914c700 1 -- 172.21.15.195:0/376813505 wait complete. 2023-10-12T22:34:47.288 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:48.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:48 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:46.828565+0000 mgr.smithi131.efktma (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:48.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:48 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:47.284340+0000 mon.smithi131 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.195:0/376813505' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:48.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:48.663 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:49.807 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.804+0000 7f1406fe2700 1 -- 172.21.15.195:0/3472906952 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14000f5410 msgr2=0x7f14000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:49.807 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.804+0000 7f1406fe2700 1 --2- 172.21.15.195:0/3472906952 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14000f5410 0x7f14000f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f13f4009fb0 tx=0x7f13f4009b30).stop 2023-10-12T22:34:49.808 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.804+0000 7f1406fe2700 1 -- 172.21.15.195:0/3472906952 shutdown_connections 2023-10-12T22:34:49.808 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.804+0000 7f1406fe2700 1 --2- 172.21.15.195:0/3472906952 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14000f5410 0x7f14000f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:49.808 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.804+0000 7f1406fe2700 1 -- 172.21.15.195:0/3472906952 >> 172.21.15.195:0/3472906952 conn(0x7f14000f0b70 msgr2=0x7f14000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:49.808 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.804+0000 7f1406fe2700 1 -- 172.21.15.195:0/3472906952 shutdown_connections 2023-10-12T22:34:49.808 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.804+0000 7f1406fe2700 1 -- 172.21.15.195:0/3472906952 wait complete. 2023-10-12T22:34:49.809 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.805+0000 7f1406fe2700 1 Processor -- start 2023-10-12T22:34:49.809 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.805+0000 7f1406fe2700 1 -- start start 2023-10-12T22:34:49.809 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.805+0000 7f1406fe2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14000f5410 0x7f1400109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:49.809 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.805+0000 7f1406fe2700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1400060220 con 0x7f14000f5410 2023-10-12T22:34:49.809 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.805+0000 7f1404d7e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14000f5410 0x7f1400109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:49.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.805+0000 7f1404d7e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14000f5410 0x7f1400109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:36608/0 (socket says 172.21.15.195:36608) 2023-10-12T22:34:49.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.806+0000 7f1404d7e700 1 -- 172.21.15.195:0/804207335 learned_addr learned my addr 172.21.15.195:0/804207335 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:49.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.806+0000 7f1404d7e700 1 -- 172.21.15.195:0/804207335 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f4005040 con 0x7f14000f5410 2023-10-12T22:34:49.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.806+0000 7f1404d7e700 1 --2- 172.21.15.195:0/804207335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14000f5410 0x7f1400109450 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f13f4013510 tx=0x7f13f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:49.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.806+0000 7f13fd7fa700 1 -- 172.21.15.195:0/804207335 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13f4009320 con 0x7f14000f5410 2023-10-12T22:34:49.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.806+0000 7f13fd7fa700 1 -- 172.21.15.195:0/804207335 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13f4013f00 con 0x7f14000f5410 2023-10-12T22:34:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.806+0000 7f1406fe2700 1 -- 172.21.15.195:0/804207335 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1400060420 con 0x7f14000f5410 2023-10-12T22:34:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.806+0000 7f13fd7fa700 1 -- 172.21.15.195:0/804207335 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13f4014d90 con 0x7f14000f5410 2023-10-12T22:34:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.806+0000 7f1406fe2700 1 -- 172.21.15.195:0/804207335 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1400109b00 con 0x7f14000f5410 2023-10-12T22:34:49.812 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.807+0000 7f13fd7fa700 1 -- 172.21.15.195:0/804207335 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13f4014420 con 0x7f14000f5410 2023-10-12T22:34:49.812 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.807+0000 7f1406fe2700 1 -- 172.21.15.195:0/804207335 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1400059100 con 0x7f14000f5410 2023-10-12T22:34:49.812 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.807+0000 7f13fd7fa700 1 --2- 172.21.15.195:0/804207335 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f13f002ff60 0x7f13f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:49.813 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.807+0000 7f13fd7fa700 1 -- 172.21.15.195:0/804207335 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13f40410c0 con 0x7f14000f5410 2023-10-12T22:34:49.813 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.807+0000 7f13fffff700 1 --2- 172.21.15.195:0/804207335 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f13f002ff60 0x7f13f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:49.813 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.808+0000 7f13fffff700 1 --2- 172.21.15.195:0/804207335 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f13f002ff60 0x7f13f0032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f13ec0096a0 tx=0x7f13ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:49.813 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:49.812+0000 7f13fd7fa700 1 -- 172.21.15.195:0/804207335 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13f40095f0 con 0x7f14000f5410 2023-10-12T22:34:50.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:50.019+0000 7f1406fe2700 1 -- 172.21.15.195:0/804207335 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1400045ad0 con 0x7f14000f5410 2023-10-12T22:34:50.021 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:50.020+0000 7f13fd7fa700 1 -- 172.21.15.195:0/804207335 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f13f401db70 con 0x7f14000f5410 2023-10-12T22:34:50.021 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:50.022 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:50.023 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:50.021+0000 7f1406fe2700 1 -- 172.21.15.195:0/804207335 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f13f002ff60 msgr2=0x7f13f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:50.023 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:50.021+0000 7f1406fe2700 1 --2- 172.21.15.195:0/804207335 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f13f002ff60 0x7f13f0032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f13ec0096a0 tx=0x7f13ec006b40).stop 2023-10-12T22:34:50.023 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:50.021+0000 7f1406fe2700 1 -- 172.21.15.195:0/804207335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14000f5410 msgr2=0x7f1400109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:50.023 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:50.021+0000 7f1406fe2700 1 --2- 172.21.15.195:0/804207335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14000f5410 0x7f1400109450 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f13f4013510 tx=0x7f13f400b040).stop 2023-10-12T22:34:50.023 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:50.022+0000 7f1406fe2700 1 -- 172.21.15.195:0/804207335 shutdown_connections 2023-10-12T22:34:50.024 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:50.022+0000 7f1406fe2700 1 --2- 172.21.15.195:0/804207335 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f13f002ff60 0x7f13f0032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:50.024 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:50.022+0000 7f1406fe2700 1 --2- 172.21.15.195:0/804207335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14000f5410 0x7f1400109450 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:50.024 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:50.022+0000 7f1406fe2700 1 -- 172.21.15.195:0/804207335 >> 172.21.15.195:0/804207335 conn(0x7f14000f0b70 msgr2=0x7f14000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:50.024 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:50.022+0000 7f1406fe2700 1 -- 172.21.15.195:0/804207335 shutdown_connections 2023-10-12T22:34:50.024 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:50.022+0000 7f1406fe2700 1 -- 172.21.15.195:0/804207335 wait complete. 2023-10-12T22:34:50.025 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:50.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:48.828863+0000 mgr.smithi131.efktma (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:50.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:50.021200+0000 mon.smithi131 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.195:0/804207335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:51.473 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:51.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:52.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:50.829232+0000 mgr.smithi131.efktma (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:52.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.636+0000 7fb79623d700 1 -- 172.21.15.195:0/3107772805 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7900f5410 msgr2=0x7fb7900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:52.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.636+0000 7fb79623d700 1 --2- 172.21.15.195:0/3107772805 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7900f5410 0x7fb7900f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fb784009fb0 tx=0x7fb784009b30).stop 2023-10-12T22:34:52.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.636+0000 7fb79623d700 1 -- 172.21.15.195:0/3107772805 shutdown_connections 2023-10-12T22:34:52.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.636+0000 7fb79623d700 1 --2- 172.21.15.195:0/3107772805 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7900f5410 0x7fb7900f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:52.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.636+0000 7fb79623d700 1 -- 172.21.15.195:0/3107772805 >> 172.21.15.195:0/3107772805 conn(0x7fb7900f0b70 msgr2=0x7fb7900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:52.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.636+0000 7fb79623d700 1 -- 172.21.15.195:0/3107772805 shutdown_connections 2023-10-12T22:34:52.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.636+0000 7fb79623d700 1 -- 172.21.15.195:0/3107772805 wait complete. 2023-10-12T22:34:52.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.637+0000 7fb79623d700 1 Processor -- start 2023-10-12T22:34:52.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.637+0000 7fb79623d700 1 -- start start 2023-10-12T22:34:52.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.637+0000 7fb79623d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7900f5410 0x7fb790108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:52.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.637+0000 7fb79623d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb790108e60 con 0x7fb7900f5410 2023-10-12T22:34:52.641 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.637+0000 7fb78f7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7900f5410 0x7fb790108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:52.642 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.637+0000 7fb78f7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7900f5410 0x7fb790108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:36628/0 (socket says 172.21.15.195:36628) 2023-10-12T22:34:52.642 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.637+0000 7fb78f7fe700 1 -- 172.21.15.195:0/637603651 learned_addr learned my addr 172.21.15.195:0/637603651 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:52.642 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.637+0000 7fb78f7fe700 1 -- 172.21.15.195:0/637603651 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb784005040 con 0x7fb7900f5410 2023-10-12T22:34:52.642 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.638+0000 7fb78f7fe700 1 --2- 172.21.15.195:0/637603651 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7900f5410 0x7fb790108970 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fb784009f80 tx=0x7fb78400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:52.642 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.638+0000 7fb777fff700 1 -- 172.21.15.195:0/637603651 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb784009320 con 0x7fb7900f5410 2023-10-12T22:34:52.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.638+0000 7fb79623d700 1 -- 172.21.15.195:0/637603651 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb790109060 con 0x7fb7900f5410 2023-10-12T22:34:52.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.638+0000 7fb777fff700 1 -- 172.21.15.195:0/637603651 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb784013ec0 con 0x7fb7900f5410 2023-10-12T22:34:52.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.638+0000 7fb777fff700 1 -- 172.21.15.195:0/637603651 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb78401ece0 con 0x7fb7900f5410 2023-10-12T22:34:52.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.638+0000 7fb79623d700 1 -- 172.21.15.195:0/637603651 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7901094f0 con 0x7fb7900f5410 2023-10-12T22:34:52.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.639+0000 7fb777fff700 1 -- 172.21.15.195:0/637603651 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb78401e420 con 0x7fb7900f5410 2023-10-12T22:34:52.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.639+0000 7fb777fff700 1 --2- 172.21.15.195:0/637603651 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb77802ff60 0x7fb778032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:52.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.639+0000 7fb79623d700 1 -- 172.21.15.195:0/637603651 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb790045ad0 con 0x7fb7900f5410 2023-10-12T22:34:52.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.639+0000 7fb777fff700 1 -- 172.21.15.195:0/637603651 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb784040e40 con 0x7fb7900f5410 2023-10-12T22:34:52.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.639+0000 7fb78effd700 1 --2- 172.21.15.195:0/637603651 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb77802ff60 0x7fb778032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:52.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.639+0000 7fb78effd700 1 --2- 172.21.15.195:0/637603651 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb77802ff60 0x7fb778032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fb7800096a0 tx=0x7fb780006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:52.645 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.643+0000 7fb777fff700 1 -- 172.21.15.195:0/637603651 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb78401e730 con 0x7fb7900f5410 2023-10-12T22:34:52.852 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.850+0000 7fb79623d700 1 -- 172.21.15.195:0/637603651 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb79010a3a0 con 0x7fb7900f5410 2023-10-12T22:34:52.852 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.851+0000 7fb777fff700 1 -- 172.21.15.195:0/637603651 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb784004490 con 0x7fb7900f5410 2023-10-12T22:34:52.853 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:52.853 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:52.854 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.853+0000 7fb79623d700 1 -- 172.21.15.195:0/637603651 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb77802ff60 msgr2=0x7fb778032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:52.854 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.853+0000 7fb79623d700 1 --2- 172.21.15.195:0/637603651 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb77802ff60 0x7fb778032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fb7800096a0 tx=0x7fb780006b40).stop 2023-10-12T22:34:52.854 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.853+0000 7fb79623d700 1 -- 172.21.15.195:0/637603651 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7900f5410 msgr2=0x7fb790108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:52.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.853+0000 7fb79623d700 1 --2- 172.21.15.195:0/637603651 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7900f5410 0x7fb790108970 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fb784009f80 tx=0x7fb78400b040).stop 2023-10-12T22:34:52.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.853+0000 7fb79623d700 1 -- 172.21.15.195:0/637603651 shutdown_connections 2023-10-12T22:34:52.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.853+0000 7fb79623d700 1 --2- 172.21.15.195:0/637603651 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb77802ff60 0x7fb778032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:52.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.853+0000 7fb79623d700 1 --2- 172.21.15.195:0/637603651 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7900f5410 0x7fb790108970 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:52.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.853+0000 7fb79623d700 1 -- 172.21.15.195:0/637603651 >> 172.21.15.195:0/637603651 conn(0x7fb7900f0b70 msgr2=0x7fb79005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:52.856 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.853+0000 7fb79623d700 1 -- 172.21.15.195:0/637603651 shutdown_connections 2023-10-12T22:34:52.856 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:52.853+0000 7fb79623d700 1 -- 172.21.15.195:0/637603651 wait complete. 2023-10-12T22:34:52.857 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:53.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:53 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:52.852595+0000 mon.smithi131 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.195:0/637603651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:54.296 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:54.296 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:54.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:52.829556+0000 mgr.smithi131.efktma (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:55.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.498+0000 7fe342dcb700 1 -- 172.21.15.195:0/3681201285 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe33c0f62a0 msgr2=0x7fe33c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:55.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.498+0000 7fe342dcb700 1 --2- 172.21.15.195:0/3681201285 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe33c0f62a0 0x7fe33c0f66c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fe32c009fb0 tx=0x7fe32c009b30).stop 2023-10-12T22:34:55.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.498+0000 7fe342dcb700 1 -- 172.21.15.195:0/3681201285 shutdown_connections 2023-10-12T22:34:55.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.498+0000 7fe342dcb700 1 --2- 172.21.15.195:0/3681201285 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe33c0f62a0 0x7fe33c0f66c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:55.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.498+0000 7fe342dcb700 1 -- 172.21.15.195:0/3681201285 >> 172.21.15.195:0/3681201285 conn(0x7fe33c0f1aa0 msgr2=0x7fe33c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:55.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.498+0000 7fe342dcb700 1 -- 172.21.15.195:0/3681201285 shutdown_connections 2023-10-12T22:34:55.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.498+0000 7fe342dcb700 1 -- 172.21.15.195:0/3681201285 wait complete. 2023-10-12T22:34:55.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.498+0000 7fe342dcb700 1 Processor -- start 2023-10-12T22:34:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.499+0000 7fe342dcb700 1 -- start start 2023-10-12T22:34:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.499+0000 7fe342dcb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe33c0f62a0 0x7fe33c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.499+0000 7fe342dcb700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c1069a0 con 0x7fe33c0f62a0 2023-10-12T22:34:55.504 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.499+0000 7fe341dc9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe33c0f62a0 0x7fe33c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:55.505 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.499+0000 7fe341dc9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe33c0f62a0 0x7fe33c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:36640/0 (socket says 172.21.15.195:36640) 2023-10-12T22:34:55.505 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.499+0000 7fe341dc9700 1 -- 172.21.15.195:0/1086593862 learned_addr learned my addr 172.21.15.195:0/1086593862 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:55.505 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.499+0000 7fe341dc9700 1 -- 172.21.15.195:0/1086593862 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c005040 con 0x7fe33c0f62a0 2023-10-12T22:34:55.505 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.499+0000 7fe341dc9700 1 --2- 172.21.15.195:0/1086593862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe33c0f62a0 0x7fe33c1064b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fe32c009830 tx=0x7fe32c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:55.505 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.500+0000 7fe3327fc700 1 -- 172.21.15.195:0/1086593862 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe32c009320 con 0x7fe33c0f62a0 2023-10-12T22:34:55.506 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.500+0000 7fe3327fc700 1 -- 172.21.15.195:0/1086593862 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe32c0136c0 con 0x7fe33c0f62a0 2023-10-12T22:34:55.506 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.500+0000 7fe342dcb700 1 -- 172.21.15.195:0/1086593862 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33c106ba0 con 0x7fe33c0f62a0 2023-10-12T22:34:55.506 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.500+0000 7fe3327fc700 1 -- 172.21.15.195:0/1086593862 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe32c01ee90 con 0x7fe33c0f62a0 2023-10-12T22:34:55.506 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.500+0000 7fe342dcb700 1 -- 172.21.15.195:0/1086593862 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33c107030 con 0x7fe33c0f62a0 2023-10-12T22:34:55.506 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.500+0000 7fe3327fc700 1 -- 172.21.15.195:0/1086593862 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe32c014440 con 0x7fe33c0f62a0 2023-10-12T22:34:55.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.501+0000 7fe3327fc700 1 --2- 172.21.15.195:0/1086593862 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe32802ff10 0x7fe3280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:55.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.501+0000 7fe3327fc700 1 -- 172.21.15.195:0/1086593862 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe32c040fe0 con 0x7fe33c0f62a0 2023-10-12T22:34:55.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.501+0000 7fe342dcb700 1 -- 172.21.15.195:0/1086593862 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33c045ad0 con 0x7fe33c0f62a0 2023-10-12T22:34:55.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.501+0000 7fe3415c8700 1 --2- 172.21.15.195:0/1086593862 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe32802ff10 0x7fe3280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:55.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.501+0000 7fe3415c8700 1 --2- 172.21.15.195:0/1086593862 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe32802ff10 0x7fe3280323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fe3380096a0 tx=0x7fe338006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:55.508 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.505+0000 7fe3327fc700 1 -- 172.21.15.195:0/1086593862 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe32c0273f0 con 0x7fe33c0f62a0 2023-10-12T22:34:55.711 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.710+0000 7fe342dcb700 1 -- 172.21.15.195:0/1086593862 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe33c107ee0 con 0x7fe33c0f62a0 2023-10-12T22:34:55.712 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.711+0000 7fe3327fc700 1 -- 172.21.15.195:0/1086593862 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe32c01c320 con 0x7fe33c0f62a0 2023-10-12T22:34:55.712 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:55.712 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:55.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.712+0000 7fe342dcb700 1 -- 172.21.15.195:0/1086593862 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe32802ff10 msgr2=0x7fe3280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:55.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.712+0000 7fe342dcb700 1 --2- 172.21.15.195:0/1086593862 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe32802ff10 0x7fe3280323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fe3380096a0 tx=0x7fe338006b40).stop 2023-10-12T22:34:55.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.712+0000 7fe342dcb700 1 -- 172.21.15.195:0/1086593862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe33c0f62a0 msgr2=0x7fe33c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:55.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.712+0000 7fe342dcb700 1 --2- 172.21.15.195:0/1086593862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe33c0f62a0 0x7fe33c1064b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fe32c009830 tx=0x7fe32c00b040).stop 2023-10-12T22:34:55.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.712+0000 7fe342dcb700 1 -- 172.21.15.195:0/1086593862 shutdown_connections 2023-10-12T22:34:55.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.712+0000 7fe342dcb700 1 --2- 172.21.15.195:0/1086593862 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe32802ff10 0x7fe3280323d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:55.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.712+0000 7fe342dcb700 1 --2- 172.21.15.195:0/1086593862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe33c0f62a0 0x7fe33c1064b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:55.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.712+0000 7fe342dcb700 1 -- 172.21.15.195:0/1086593862 >> 172.21.15.195:0/1086593862 conn(0x7fe33c0f1aa0 msgr2=0x7fe33c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:55.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.712+0000 7fe342dcb700 1 -- 172.21.15.195:0/1086593862 shutdown_connections 2023-10-12T22:34:55.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:55.712+0000 7fe342dcb700 1 -- 172.21.15.195:0/1086593862 wait complete. 2023-10-12T22:34:55.716 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:56.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:54.829927+0000 mgr.smithi131.efktma (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:56.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:55.712069+0000 mon.smithi131 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.195:0/1086593862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:57.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:57.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:34:58.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.227+0000 7fb9e6068700 1 -- 172.21.15.195:0/4166141432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9e00f5410 msgr2=0x7fb9e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:58.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.227+0000 7fb9e6068700 1 --2- 172.21.15.195:0/4166141432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9e00f5410 0x7fb9e00f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fb9d4009fb0 tx=0x7fb9d4009b30).stop 2023-10-12T22:34:58.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.227+0000 7fb9e6068700 1 -- 172.21.15.195:0/4166141432 shutdown_connections 2023-10-12T22:34:58.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.227+0000 7fb9e6068700 1 --2- 172.21.15.195:0/4166141432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9e00f5410 0x7fb9e00f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:58.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.227+0000 7fb9e6068700 1 -- 172.21.15.195:0/4166141432 >> 172.21.15.195:0/4166141432 conn(0x7fb9e00f0b70 msgr2=0x7fb9e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:58.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.227+0000 7fb9e6068700 1 -- 172.21.15.195:0/4166141432 shutdown_connections 2023-10-12T22:34:58.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.227+0000 7fb9e6068700 1 -- 172.21.15.195:0/4166141432 wait complete. 2023-10-12T22:34:58.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.228+0000 7fb9e6068700 1 Processor -- start 2023-10-12T22:34:58.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.228+0000 7fb9e6068700 1 -- start start 2023-10-12T22:34:58.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.228+0000 7fb9e6068700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9e00f5410 0x7fb9e0105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:58.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.228+0000 7fb9e6068700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e0105af0 con 0x7fb9e00f5410 2023-10-12T22:34:58.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.229+0000 7fb9df7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9e00f5410 0x7fb9e0105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:58.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.229+0000 7fb9df7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9e00f5410 0x7fb9e0105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:36660/0 (socket says 172.21.15.195:36660) 2023-10-12T22:34:58.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.229+0000 7fb9df7fe700 1 -- 172.21.15.195:0/285359251 learned_addr learned my addr 172.21.15.195:0/285359251 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:34:58.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.229+0000 7fb9df7fe700 1 -- 172.21.15.195:0/285359251 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d4005040 con 0x7fb9e00f5410 2023-10-12T22:34:58.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.229+0000 7fb9df7fe700 1 --2- 172.21.15.195:0/285359251 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9e00f5410 0x7fb9e0105600 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fb9d4014000 tx=0x7fb9d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:58.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.229+0000 7fb9c7fff700 1 -- 172.21.15.195:0/285359251 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9d40095f0 con 0x7fb9e00f5410 2023-10-12T22:34:58.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.229+0000 7fb9e6068700 1 -- 172.21.15.195:0/285359251 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e0105cf0 con 0x7fb9e00f5410 2023-10-12T22:34:58.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.229+0000 7fb9c7fff700 1 -- 172.21.15.195:0/285359251 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9d4007e70 con 0x7fb9e00f5410 2023-10-12T22:34:58.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.229+0000 7fb9c7fff700 1 -- 172.21.15.195:0/285359251 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9d4014ea0 con 0x7fb9e00f5410 2023-10-12T22:34:58.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.229+0000 7fb9e6068700 1 -- 172.21.15.195:0/285359251 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e0106180 con 0x7fb9e00f5410 2023-10-12T22:34:58.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.230+0000 7fb9c7fff700 1 -- 172.21.15.195:0/285359251 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9d40079c0 con 0x7fb9e00f5410 2023-10-12T22:34:58.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.230+0000 7fb9e6068700 1 -- 172.21.15.195:0/285359251 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e0045ad0 con 0x7fb9e00f5410 2023-10-12T22:34:58.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.230+0000 7fb9c7fff700 1 --2- 172.21.15.195:0/285359251 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb9c802ffb0 0x7fb9c8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:34:58.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.230+0000 7fb9c7fff700 1 -- 172.21.15.195:0/285359251 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9d4040920 con 0x7fb9e00f5410 2023-10-12T22:34:58.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.231+0000 7fb9deffd700 1 --2- 172.21.15.195:0/285359251 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb9c802ffb0 0x7fb9c8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:34:58.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.231+0000 7fb9deffd700 1 --2- 172.21.15.195:0/285359251 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb9c802ffb0 0x7fb9c8032470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fb9d00096a0 tx=0x7fb9d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:34:58.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.235+0000 7fb9c7fff700 1 -- 172.21.15.195:0/285359251 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9d400f950 con 0x7fb9e00f5410 2023-10-12T22:34:58.443 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.442+0000 7fb9e6068700 1 -- 172.21.15.195:0/285359251 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9e00fae60 con 0x7fb9e00f5410 2023-10-12T22:34:58.445 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.443+0000 7fb9c7fff700 1 -- 172.21.15.195:0/285359251 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb9d4010230 con 0x7fb9e00f5410 2023-10-12T22:34:58.445 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:34:58.445 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:34:58.446 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.445+0000 7fb9e6068700 1 -- 172.21.15.195:0/285359251 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb9c802ffb0 msgr2=0x7fb9c8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:58.447 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.445+0000 7fb9e6068700 1 --2- 172.21.15.195:0/285359251 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb9c802ffb0 0x7fb9c8032470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fb9d00096a0 tx=0x7fb9d0006b40).stop 2023-10-12T22:34:58.447 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.445+0000 7fb9e6068700 1 -- 172.21.15.195:0/285359251 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9e00f5410 msgr2=0x7fb9e0105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:34:58.447 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.445+0000 7fb9e6068700 1 --2- 172.21.15.195:0/285359251 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9e00f5410 0x7fb9e0105600 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fb9d4014000 tx=0x7fb9d400b040).stop 2023-10-12T22:34:58.447 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.445+0000 7fb9e6068700 1 -- 172.21.15.195:0/285359251 shutdown_connections 2023-10-12T22:34:58.448 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.445+0000 7fb9e6068700 1 --2- 172.21.15.195:0/285359251 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb9c802ffb0 0x7fb9c8032470 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:58.448 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.445+0000 7fb9e6068700 1 --2- 172.21.15.195:0/285359251 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9e00f5410 0x7fb9e0105600 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:34:58.448 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.445+0000 7fb9e6068700 1 -- 172.21.15.195:0/285359251 >> 172.21.15.195:0/285359251 conn(0x7fb9e00f0b70 msgr2=0x7fb9e00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:34:58.449 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.445+0000 7fb9e6068700 1 -- 172.21.15.195:0/285359251 shutdown_connections 2023-10-12T22:34:58.449 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:34:58.445+0000 7fb9e6068700 1 -- 172.21.15.195:0/285359251 wait complete. 2023-10-12T22:34:58.449 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:34:58.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:58 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:56.830244+0000 mgr.smithi131.efktma (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:34:59.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:34:59 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:34:58.444335+0000 mon.smithi131 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.195:0/285359251' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:34:59.840 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:34:59.841 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:00.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:34:58.830595+0000 mgr.smithi131.efktma (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:01.008 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.006+0000 7f5de8a54700 1 -- 172.21.15.195:0/3907455805 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5de40f6510 msgr2=0x7f5de40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:01.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.006+0000 7f5de8a54700 1 --2- 172.21.15.195:0/3907455805 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5de40f6510 0x7f5de40f6930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f5dcc009fb0 tx=0x7f5dcc009b30).stop 2023-10-12T22:35:01.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.006+0000 7f5de8a54700 1 -- 172.21.15.195:0/3907455805 shutdown_connections 2023-10-12T22:35:01.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.006+0000 7f5de8a54700 1 --2- 172.21.15.195:0/3907455805 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5de40f6510 0x7f5de40f6930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:01.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.006+0000 7f5de8a54700 1 -- 172.21.15.195:0/3907455805 >> 172.21.15.195:0/3907455805 conn(0x7f5de40f1c70 msgr2=0x7f5de40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:01.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.006+0000 7f5de8a54700 1 -- 172.21.15.195:0/3907455805 shutdown_connections 2023-10-12T22:35:01.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.006+0000 7f5de8a54700 1 -- 172.21.15.195:0/3907455805 wait complete. 2023-10-12T22:35:01.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.007+0000 7f5de8a54700 1 Processor -- start 2023-10-12T22:35:01.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.007+0000 7f5de8a54700 1 -- start start 2023-10-12T22:35:01.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.007+0000 7f5de8a54700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5de40f6510 0x7f5de406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:01.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.007+0000 7f5de8a54700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de4069e20 con 0x7f5de40f6510 2023-10-12T22:35:01.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.007+0000 7f5de259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5de40f6510 0x7f5de406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:01.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.007+0000 7f5de259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5de40f6510 0x7f5de406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38120/0 (socket says 172.21.15.195:38120) 2023-10-12T22:35:01.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.007+0000 7f5de259c700 1 -- 172.21.15.195:0/3076515268 learned_addr learned my addr 172.21.15.195:0/3076515268 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:01.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.007+0000 7f5de259c700 1 -- 172.21.15.195:0/3076515268 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc005040 con 0x7f5de40f6510 2023-10-12T22:35:01.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.008+0000 7f5de259c700 1 --2- 172.21.15.195:0/3076515268 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5de40f6510 0x7f5de406b9a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f5dcc013510 tx=0x7f5dcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:01.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.008+0000 7f5ddaffd700 1 -- 172.21.15.195:0/3076515268 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5dcc009320 con 0x7f5de40f6510 2023-10-12T22:35:01.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.008+0000 7f5ddaffd700 1 -- 172.21.15.195:0/3076515268 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5dcc013f00 con 0x7f5de40f6510 2023-10-12T22:35:01.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.008+0000 7f5de8a54700 1 -- 172.21.15.195:0/3076515268 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de4069fc0 con 0x7f5de40f6510 2023-10-12T22:35:01.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.008+0000 7f5ddaffd700 1 -- 172.21.15.195:0/3076515268 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5dcc014d90 con 0x7f5de40f6510 2023-10-12T22:35:01.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.008+0000 7f5de8a54700 1 -- 172.21.15.195:0/3076515268 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de406a450 con 0x7f5de40f6510 2023-10-12T22:35:01.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.008+0000 7f5ddaffd700 1 -- 172.21.15.195:0/3076515268 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5dcc014420 con 0x7f5de40f6510 2023-10-12T22:35:01.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.009+0000 7f5ddaffd700 1 --2- 172.21.15.195:0/3076515268 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5dd002ff60 0x7f5dd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:01.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.009+0000 7f5de8a54700 1 -- 172.21.15.195:0/3076515268 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de4059100 con 0x7f5de40f6510 2023-10-12T22:35:01.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.009+0000 7f5ddaffd700 1 -- 172.21.15.195:0/3076515268 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5dcc041000 con 0x7f5de40f6510 2023-10-12T22:35:01.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.009+0000 7f5de1d9b700 1 --2- 172.21.15.195:0/3076515268 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5dd002ff60 0x7f5dd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:01.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.009+0000 7f5de1d9b700 1 --2- 172.21.15.195:0/3076515268 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5dd002ff60 0x7f5dd0032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f5dd40096a0 tx=0x7f5dd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:01.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.013+0000 7f5ddaffd700 1 -- 172.21.15.195:0/3076515268 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5dcc0095f0 con 0x7f5de40f6510 2023-10-12T22:35:01.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.226+0000 7f5de8a54700 1 -- 172.21.15.195:0/3076515268 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5de40fbf60 con 0x7f5de40f6510 2023-10-12T22:35:01.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.227+0000 7f5ddaffd700 1 -- 172.21.15.195:0/3076515268 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5dcc01db70 con 0x7f5de40f6510 2023-10-12T22:35:01.231 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:01.231 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:01.232 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.229+0000 7f5de8a54700 1 -- 172.21.15.195:0/3076515268 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5dd002ff60 msgr2=0x7f5dd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:01.232 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.229+0000 7f5de8a54700 1 --2- 172.21.15.195:0/3076515268 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5dd002ff60 0x7f5dd0032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f5dd40096a0 tx=0x7f5dd4006b40).stop 2023-10-12T22:35:01.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.229+0000 7f5de8a54700 1 -- 172.21.15.195:0/3076515268 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5de40f6510 msgr2=0x7f5de406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:01.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.229+0000 7f5de8a54700 1 --2- 172.21.15.195:0/3076515268 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5de40f6510 0x7f5de406b9a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f5dcc013510 tx=0x7f5dcc00b040).stop 2023-10-12T22:35:01.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.229+0000 7f5de8a54700 1 -- 172.21.15.195:0/3076515268 shutdown_connections 2023-10-12T22:35:01.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.229+0000 7f5de8a54700 1 --2- 172.21.15.195:0/3076515268 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5dd002ff60 0x7f5dd0032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:01.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.229+0000 7f5de8a54700 1 --2- 172.21.15.195:0/3076515268 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5de40f6510 0x7f5de406b9a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:01.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.229+0000 7f5de8a54700 1 -- 172.21.15.195:0/3076515268 >> 172.21.15.195:0/3076515268 conn(0x7f5de40f1c70 msgr2=0x7f5de40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:01.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.229+0000 7f5de8a54700 1 -- 172.21.15.195:0/3076515268 shutdown_connections 2023-10-12T22:35:01.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:01.229+0000 7f5de8a54700 1 -- 172.21.15.195:0/3076515268 wait complete. 2023-10-12T22:35:01.234 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:02.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:02 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:00.830935+0000 mgr.smithi131.efktma (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:02.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:02 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:01.228426+0000 mon.smithi131 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.195:0/3076515268' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:02.602 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:02.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:03.762 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.760+0000 7fdf9a810700 1 -- 172.21.15.195:0/2537489431 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf940f5410 msgr2=0x7fdf940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:03.762 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.760+0000 7fdf9a810700 1 --2- 172.21.15.195:0/2537489431 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf940f5410 0x7fdf940f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fdf84009fb0 tx=0x7fdf84009b30).stop 2023-10-12T22:35:03.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.760+0000 7fdf9a810700 1 -- 172.21.15.195:0/2537489431 shutdown_connections 2023-10-12T22:35:03.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.760+0000 7fdf9a810700 1 --2- 172.21.15.195:0/2537489431 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf940f5410 0x7fdf940f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:03.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.760+0000 7fdf9a810700 1 -- 172.21.15.195:0/2537489431 >> 172.21.15.195:0/2537489431 conn(0x7fdf940f0b70 msgr2=0x7fdf940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:03.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.760+0000 7fdf9a810700 1 -- 172.21.15.195:0/2537489431 shutdown_connections 2023-10-12T22:35:03.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.760+0000 7fdf9a810700 1 -- 172.21.15.195:0/2537489431 wait complete. 2023-10-12T22:35:03.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.761+0000 7fdf9a810700 1 Processor -- start 2023-10-12T22:35:03.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.761+0000 7fdf9a810700 1 -- start start 2023-10-12T22:35:03.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.761+0000 7fdf9a810700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf940f5410 0x7fdf94105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:03.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.761+0000 7fdf9a810700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf94105b20 con 0x7fdf940f5410 2023-10-12T22:35:03.766 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.761+0000 7fdf93fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf940f5410 0x7fdf94105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:03.766 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.761+0000 7fdf93fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf940f5410 0x7fdf94105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38138/0 (socket says 172.21.15.195:38138) 2023-10-12T22:35:03.766 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.761+0000 7fdf93fff700 1 -- 172.21.15.195:0/1595938353 learned_addr learned my addr 172.21.15.195:0/1595938353 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:03.766 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.762+0000 7fdf93fff700 1 -- 172.21.15.195:0/1595938353 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf84005040 con 0x7fdf940f5410 2023-10-12T22:35:03.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.762+0000 7fdf93fff700 1 --2- 172.21.15.195:0/1595938353 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf940f5410 0x7fdf94105630 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fdf84013510 tx=0x7fdf8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:03.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.762+0000 7fdf90ff9700 1 -- 172.21.15.195:0/1595938353 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf84009320 con 0x7fdf940f5410 2023-10-12T22:35:03.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.762+0000 7fdf9a810700 1 -- 172.21.15.195:0/1595938353 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf94105d20 con 0x7fdf940f5410 2023-10-12T22:35:03.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.762+0000 7fdf90ff9700 1 -- 172.21.15.195:0/1595938353 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf84013f00 con 0x7fdf940f5410 2023-10-12T22:35:03.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.762+0000 7fdf90ff9700 1 -- 172.21.15.195:0/1595938353 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf84014d90 con 0x7fdf940f5410 2023-10-12T22:35:03.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.762+0000 7fdf9a810700 1 -- 172.21.15.195:0/1595938353 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf941061b0 con 0x7fdf940f5410 2023-10-12T22:35:03.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.763+0000 7fdf90ff9700 1 -- 172.21.15.195:0/1595938353 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf84014420 con 0x7fdf940f5410 2023-10-12T22:35:03.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.763+0000 7fdf90ff9700 1 --2- 172.21.15.195:0/1595938353 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf8002ff60 0x7fdf80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:03.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.763+0000 7fdf9a810700 1 -- 172.21.15.195:0/1595938353 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf94045ad0 con 0x7fdf940f5410 2023-10-12T22:35:03.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.763+0000 7fdf90ff9700 1 -- 172.21.15.195:0/1595938353 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf840410c0 con 0x7fdf940f5410 2023-10-12T22:35:03.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.763+0000 7fdf937fe700 1 --2- 172.21.15.195:0/1595938353 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf8002ff60 0x7fdf80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:03.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.764+0000 7fdf937fe700 1 --2- 172.21.15.195:0/1595938353 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf8002ff60 0x7fdf80032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fdf7c0096a0 tx=0x7fdf7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:03.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.768+0000 7fdf90ff9700 1 -- 172.21.15.195:0/1595938353 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf8406d2b0 con 0x7fdf940f5410 2023-10-12T22:35:03.976 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.975+0000 7fdf9a810700 1 -- 172.21.15.195:0/1595938353 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf940faf50 con 0x7fdf940f5410 2023-10-12T22:35:03.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.975+0000 7fdf90ff9700 1 -- 172.21.15.195:0/1595938353 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdf8401dd50 con 0x7fdf940f5410 2023-10-12T22:35:03.977 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:03.977 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:03.978 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.977+0000 7fdf9a810700 1 -- 172.21.15.195:0/1595938353 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf8002ff60 msgr2=0x7fdf80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:03.979 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.977+0000 7fdf9a810700 1 --2- 172.21.15.195:0/1595938353 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf8002ff60 0x7fdf80032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fdf7c0096a0 tx=0x7fdf7c006b40).stop 2023-10-12T22:35:03.981 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.977+0000 7fdf9a810700 1 -- 172.21.15.195:0/1595938353 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf940f5410 msgr2=0x7fdf94105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:03.981 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.977+0000 7fdf9a810700 1 --2- 172.21.15.195:0/1595938353 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf940f5410 0x7fdf94105630 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fdf84013510 tx=0x7fdf8400b040).stop 2023-10-12T22:35:03.981 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.977+0000 7fdf9a810700 1 -- 172.21.15.195:0/1595938353 shutdown_connections 2023-10-12T22:35:03.981 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.977+0000 7fdf9a810700 1 --2- 172.21.15.195:0/1595938353 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf8002ff60 0x7fdf80032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:03.982 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.977+0000 7fdf9a810700 1 --2- 172.21.15.195:0/1595938353 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf940f5410 0x7fdf94105630 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:03.982 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.977+0000 7fdf9a810700 1 -- 172.21.15.195:0/1595938353 >> 172.21.15.195:0/1595938353 conn(0x7fdf940f0b70 msgr2=0x7fdf940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:03.983 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.977+0000 7fdf9a810700 1 -- 172.21.15.195:0/1595938353 shutdown_connections 2023-10-12T22:35:03.983 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:03.977+0000 7fdf9a810700 1 -- 172.21.15.195:0/1595938353 wait complete. 2023-10-12T22:35:03.984 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:04.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:02.831252+0000 mgr.smithi131.efktma (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:04.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:03.976897+0000 mon.smithi131 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.195:0/1595938353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:05.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:05.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:06.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:04.831536+0000 mgr.smithi131.efktma (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:06.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.569+0000 7f135b4e9700 1 -- 172.21.15.195:0/1714789800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13540f5410 msgr2=0x7f13540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:06.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.569+0000 7f135b4e9700 1 --2- 172.21.15.195:0/1714789800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13540f5410 0x7f13540f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f1348009fb0 tx=0x7f1348009b30).stop 2023-10-12T22:35:06.573 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.569+0000 7f135b4e9700 1 -- 172.21.15.195:0/1714789800 shutdown_connections 2023-10-12T22:35:06.573 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.569+0000 7f135b4e9700 1 --2- 172.21.15.195:0/1714789800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13540f5410 0x7f13540f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:06.573 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.569+0000 7f135b4e9700 1 -- 172.21.15.195:0/1714789800 >> 172.21.15.195:0/1714789800 conn(0x7f13540f0b70 msgr2=0x7f13540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:06.573 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.569+0000 7f135b4e9700 1 -- 172.21.15.195:0/1714789800 shutdown_connections 2023-10-12T22:35:06.573 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.569+0000 7f135b4e9700 1 -- 172.21.15.195:0/1714789800 wait complete. 2023-10-12T22:35:06.574 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.569+0000 7f135b4e9700 1 Processor -- start 2023-10-12T22:35:06.574 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.570+0000 7f135b4e9700 1 -- start start 2023-10-12T22:35:06.574 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.570+0000 7f135b4e9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13540f5410 0x7f1354109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:06.574 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.570+0000 7f135b4e9700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1354060240 con 0x7f13540f5410 2023-10-12T22:35:06.574 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.570+0000 7f1359285700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13540f5410 0x7f1354109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:06.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.570+0000 7f1359285700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13540f5410 0x7f1354109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38148/0 (socket says 172.21.15.195:38148) 2023-10-12T22:35:06.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.570+0000 7f1359285700 1 -- 172.21.15.195:0/908453660 learned_addr learned my addr 172.21.15.195:0/908453660 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:06.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.570+0000 7f1359285700 1 -- 172.21.15.195:0/908453660 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1348005040 con 0x7f13540f5410 2023-10-12T22:35:06.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.570+0000 7f1359285700 1 --2- 172.21.15.195:0/908453660 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13540f5410 0x7f1354109430 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f1348013510 tx=0x7f134800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:06.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.571+0000 7f1345ffb700 1 -- 172.21.15.195:0/908453660 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1348009320 con 0x7f13540f5410 2023-10-12T22:35:06.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.571+0000 7f135b4e9700 1 -- 172.21.15.195:0/908453660 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1354060440 con 0x7f13540f5410 2023-10-12T22:35:06.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.571+0000 7f1345ffb700 1 -- 172.21.15.195:0/908453660 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1348013f00 con 0x7f13540f5410 2023-10-12T22:35:06.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.571+0000 7f1345ffb700 1 -- 172.21.15.195:0/908453660 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1348014d90 con 0x7f13540f5410 2023-10-12T22:35:06.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.571+0000 7f135b4e9700 1 -- 172.21.15.195:0/908453660 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1354109ae0 con 0x7f13540f5410 2023-10-12T22:35:06.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.572+0000 7f1345ffb700 1 -- 172.21.15.195:0/908453660 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13480095f0 con 0x7f13540f5410 2023-10-12T22:35:06.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.572+0000 7f135b4e9700 1 -- 172.21.15.195:0/908453660 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1354045ad0 con 0x7f13540f5410 2023-10-12T22:35:06.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.572+0000 7f1345ffb700 1 --2- 172.21.15.195:0/908453660 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f134002ff60 0x7f1340032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:06.578 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.572+0000 7f1345ffb700 1 -- 172.21.15.195:0/908453660 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1348041000 con 0x7f13540f5410 2023-10-12T22:35:06.578 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.572+0000 7f1358a84700 1 --2- 172.21.15.195:0/908453660 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f134002ff60 0x7f1340032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:06.578 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.573+0000 7f1358a84700 1 --2- 172.21.15.195:0/908453660 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f134002ff60 0x7f1340032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f13500096a0 tx=0x7f1350006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:06.579 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.577+0000 7f1345ffb700 1 -- 172.21.15.195:0/908453660 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f134801d7f0 con 0x7f13540f5410 2023-10-12T22:35:06.785 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.784+0000 7f135b4e9700 1 -- 172.21.15.195:0/908453660 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13540fae60 con 0x7f13540f5410 2023-10-12T22:35:06.786 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.785+0000 7f1345ffb700 1 -- 172.21.15.195:0/908453660 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1348010230 con 0x7f13540f5410 2023-10-12T22:35:06.786 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:06.787 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:06.788 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.786+0000 7f135b4e9700 1 -- 172.21.15.195:0/908453660 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f134002ff60 msgr2=0x7f1340032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:06.788 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.786+0000 7f135b4e9700 1 --2- 172.21.15.195:0/908453660 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f134002ff60 0x7f1340032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f13500096a0 tx=0x7f1350006b40).stop 2023-10-12T22:35:06.788 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.786+0000 7f135b4e9700 1 -- 172.21.15.195:0/908453660 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13540f5410 msgr2=0x7f1354109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:06.788 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.786+0000 7f135b4e9700 1 --2- 172.21.15.195:0/908453660 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13540f5410 0x7f1354109430 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f1348013510 tx=0x7f134800b040).stop 2023-10-12T22:35:06.788 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.786+0000 7f135b4e9700 1 -- 172.21.15.195:0/908453660 shutdown_connections 2023-10-12T22:35:06.789 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.786+0000 7f135b4e9700 1 --2- 172.21.15.195:0/908453660 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f134002ff60 0x7f1340032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:06.789 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.786+0000 7f135b4e9700 1 --2- 172.21.15.195:0/908453660 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f13540f5410 0x7f1354109430 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:06.789 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.786+0000 7f135b4e9700 1 -- 172.21.15.195:0/908453660 >> 172.21.15.195:0/908453660 conn(0x7f13540f0b70 msgr2=0x7f13540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:06.789 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.787+0000 7f135b4e9700 1 -- 172.21.15.195:0/908453660 shutdown_connections 2023-10-12T22:35:06.790 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:06.787+0000 7f135b4e9700 1 -- 172.21.15.195:0/908453660 wait complete. 2023-10-12T22:35:06.790 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:07.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:06.786299+0000 mon.smithi131 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.195:0/908453660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:08.177 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:08.178 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:08.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:06.831841+0000 mgr.smithi131.efktma (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:09.354 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.351+0000 7f51838b0700 1 -- 172.21.15.195:0/1705552035 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f517c0f5410 msgr2=0x7f517c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:09.354 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.351+0000 7f51838b0700 1 --2- 172.21.15.195:0/1705552035 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f517c0f5410 0x7f517c0f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f516c009fb0 tx=0x7f516c009b30).stop 2023-10-12T22:35:09.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.351+0000 7f51838b0700 1 -- 172.21.15.195:0/1705552035 shutdown_connections 2023-10-12T22:35:09.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.351+0000 7f51838b0700 1 --2- 172.21.15.195:0/1705552035 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f517c0f5410 0x7f517c0f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:09.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.351+0000 7f51838b0700 1 -- 172.21.15.195:0/1705552035 >> 172.21.15.195:0/1705552035 conn(0x7f517c0f0b70 msgr2=0x7f517c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:09.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.351+0000 7f51838b0700 1 -- 172.21.15.195:0/1705552035 shutdown_connections 2023-10-12T22:35:09.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.351+0000 7f51838b0700 1 -- 172.21.15.195:0/1705552035 wait complete. 2023-10-12T22:35:09.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.352+0000 7f51838b0700 1 Processor -- start 2023-10-12T22:35:09.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.352+0000 7f51838b0700 1 -- start start 2023-10-12T22:35:09.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.352+0000 7f51838b0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f517c0f5410 0x7f517c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:09.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.352+0000 7f51838b0700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c060220 con 0x7f517c0f5410 2023-10-12T22:35:09.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.352+0000 7f518164c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f517c0f5410 0x7f517c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:09.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.352+0000 7f518164c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f517c0f5410 0x7f517c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51632/0 (socket says 172.21.15.195:51632) 2023-10-12T22:35:09.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.352+0000 7f518164c700 1 -- 172.21.15.195:0/3516144411 learned_addr learned my addr 172.21.15.195:0/3516144411 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:09.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.353+0000 7f518164c700 1 -- 172.21.15.195:0/3516144411 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c005040 con 0x7f517c0f5410 2023-10-12T22:35:09.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.353+0000 7f518164c700 1 --2- 172.21.15.195:0/3516144411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f517c0f5410 0x7f517c109450 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f516c013510 tx=0x7f516c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:09.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.353+0000 7f5171ffb700 1 -- 172.21.15.195:0/3516144411 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f516c009320 con 0x7f517c0f5410 2023-10-12T22:35:09.358 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.353+0000 7f51838b0700 1 -- 172.21.15.195:0/3516144411 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517c060420 con 0x7f517c0f5410 2023-10-12T22:35:09.358 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.353+0000 7f5171ffb700 1 -- 172.21.15.195:0/3516144411 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f516c013f00 con 0x7f517c0f5410 2023-10-12T22:35:09.358 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.353+0000 7f5171ffb700 1 -- 172.21.15.195:0/3516144411 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f516c014d90 con 0x7f517c0f5410 2023-10-12T22:35:09.358 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.353+0000 7f51838b0700 1 -- 172.21.15.195:0/3516144411 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517c109b00 con 0x7f517c0f5410 2023-10-12T22:35:09.359 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.354+0000 7f51838b0700 1 -- 172.21.15.195:0/3516144411 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517c059100 con 0x7f517c0f5410 2023-10-12T22:35:09.359 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.354+0000 7f5171ffb700 1 -- 172.21.15.195:0/3516144411 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f516c0095f0 con 0x7f517c0f5410 2023-10-12T22:35:09.359 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.354+0000 7f5171ffb700 1 --2- 172.21.15.195:0/3516144411 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f516802ff60 0x7f5168032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:09.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.354+0000 7f5171ffb700 1 -- 172.21.15.195:0/3516144411 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f516c041000 con 0x7f517c0f5410 2023-10-12T22:35:09.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.354+0000 7f5180e4b700 1 --2- 172.21.15.195:0/3516144411 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f516802ff60 0x7f5168032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:09.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.355+0000 7f5180e4b700 1 --2- 172.21.15.195:0/3516144411 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f516802ff60 0x7f5168032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f51780096a0 tx=0x7f5178006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:09.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.359+0000 7f5171ffb700 1 -- 172.21.15.195:0/3516144411 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f516c01d7f0 con 0x7f517c0f5410 2023-10-12T22:35:09.569 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.567+0000 7f51838b0700 1 -- 172.21.15.195:0/3516144411 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f517c045ad0 con 0x7f517c0f5410 2023-10-12T22:35:09.569 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.568+0000 7f5171ffb700 1 -- 172.21.15.195:0/3516144411 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f516c010230 con 0x7f517c0f5410 2023-10-12T22:35:09.570 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:09.570 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:09.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.569+0000 7f51838b0700 1 -- 172.21.15.195:0/3516144411 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f516802ff60 msgr2=0x7f5168032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:09.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.569+0000 7f51838b0700 1 --2- 172.21.15.195:0/3516144411 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f516802ff60 0x7f5168032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f51780096a0 tx=0x7f5178006b40).stop 2023-10-12T22:35:09.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.569+0000 7f51838b0700 1 -- 172.21.15.195:0/3516144411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f517c0f5410 msgr2=0x7f517c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:09.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.569+0000 7f51838b0700 1 --2- 172.21.15.195:0/3516144411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f517c0f5410 0x7f517c109450 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f516c013510 tx=0x7f516c00b040).stop 2023-10-12T22:35:09.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.570+0000 7f51838b0700 1 -- 172.21.15.195:0/3516144411 shutdown_connections 2023-10-12T22:35:09.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.570+0000 7f51838b0700 1 --2- 172.21.15.195:0/3516144411 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f516802ff60 0x7f5168032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:09.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.570+0000 7f51838b0700 1 --2- 172.21.15.195:0/3516144411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f517c0f5410 0x7f517c109450 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:09.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.570+0000 7f51838b0700 1 -- 172.21.15.195:0/3516144411 >> 172.21.15.195:0/3516144411 conn(0x7f517c0f0b70 msgr2=0x7f517c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:09.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.570+0000 7f51838b0700 1 -- 172.21.15.195:0/3516144411 shutdown_connections 2023-10-12T22:35:09.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:09.570+0000 7f51838b0700 1 -- 172.21.15.195:0/3516144411 wait complete. 2023-10-12T22:35:09.573 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:10.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:08.832197+0000 mgr.smithi131.efktma (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:10.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:09.569441+0000 mon.smithi131 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.195:0/3516144411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:10.962 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:10.962 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:12.175 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.172+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/2172884770 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0fdc0f51a0 msgr2=0x7f0fdc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:12.176 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.172+0000 7f0fe0e7e700 1 --2- 172.21.15.195:0/2172884770 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0fdc0f51a0 0x7f0fdc0f55c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f0fcc009fb0 tx=0x7f0fcc009b30).stop 2023-10-12T22:35:12.176 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.172+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/2172884770 shutdown_connections 2023-10-12T22:35:12.176 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.172+0000 7f0fe0e7e700 1 --2- 172.21.15.195:0/2172884770 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0fdc0f51a0 0x7f0fdc0f55c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:12.176 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.172+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/2172884770 >> 172.21.15.195:0/2172884770 conn(0x7f0fdc0f09a0 msgr2=0x7f0fdc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:12.176 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.172+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/2172884770 shutdown_connections 2023-10-12T22:35:12.177 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.172+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/2172884770 wait complete. 2023-10-12T22:35:12.177 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.173+0000 7f0fe0e7e700 1 Processor -- start 2023-10-12T22:35:12.177 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.173+0000 7f0fe0e7e700 1 -- start start 2023-10-12T22:35:12.177 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.173+0000 7f0fe0e7e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0fdc0f51a0 0x7f0fdc1086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:12.177 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.173+0000 7f0fe0e7e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc108bb0 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.178 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.173+0000 7f0fdb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0fdc0f51a0 0x7f0fdc1086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:12.178 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.173+0000 7f0fdb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0fdc0f51a0 0x7f0fdc1086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51654/0 (socket says 172.21.15.195:51654) 2023-10-12T22:35:12.178 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.173+0000 7f0fdb7fe700 1 -- 172.21.15.195:0/418541858 learned_addr learned my addr 172.21.15.195:0/418541858 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:12.178 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.174+0000 7f0fdb7fe700 1 -- 172.21.15.195:0/418541858 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc005040 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.178 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.174+0000 7f0fdb7fe700 1 --2- 172.21.15.195:0/418541858 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0fdc0f51a0 0x7f0fdc1086c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f0fcc013510 tx=0x7f0fcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:12.179 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.174+0000 7f0fc3fff700 1 -- 172.21.15.195:0/418541858 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fcc009320 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.179 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.174+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/418541858 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fdc108db0 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.179 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.174+0000 7f0fc3fff700 1 -- 172.21.15.195:0/418541858 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fcc013f00 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.180 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.174+0000 7f0fc3fff700 1 -- 172.21.15.195:0/418541858 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fcc014dc0 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.180 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.174+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/418541858 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fdc109240 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.181 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.175+0000 7f0fc3fff700 1 -- 172.21.15.195:0/418541858 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0fcc014420 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.181 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.175+0000 7f0fc3fff700 1 --2- 172.21.15.195:0/418541858 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0fc402ff60 0x7f0fc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:12.181 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.175+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/418541858 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fdc045ad0 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.181 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.175+0000 7f0fc3fff700 1 -- 172.21.15.195:0/418541858 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fcc0410e0 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.182 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.175+0000 7f0fdaffd700 1 --2- 172.21.15.195:0/418541858 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0fc402ff60 0x7f0fc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:12.182 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.176+0000 7f0fdaffd700 1 --2- 172.21.15.195:0/418541858 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0fc402ff60 0x7f0fc4032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f0fd00096a0 tx=0x7f0fd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:12.182 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.180+0000 7f0fc3fff700 1 -- 172.21.15.195:0/418541858 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fcc0095f0 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.387 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.386+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/418541858 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fdc10a090 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.388 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.387+0000 7f0fc3fff700 1 -- 172.21.15.195:0/418541858 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0fcc02b020 con 0x7f0fdc0f51a0 2023-10-12T22:35:12.388 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:12.389 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:12.390 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.388+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/418541858 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0fc402ff60 msgr2=0x7f0fc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:12.390 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.388+0000 7f0fe0e7e700 1 --2- 172.21.15.195:0/418541858 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0fc402ff60 0x7f0fc4032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f0fd00096a0 tx=0x7f0fd0006b40).stop 2023-10-12T22:35:12.390 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.388+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/418541858 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0fdc0f51a0 msgr2=0x7f0fdc1086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:12.390 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.388+0000 7f0fe0e7e700 1 --2- 172.21.15.195:0/418541858 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0fdc0f51a0 0x7f0fdc1086c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f0fcc013510 tx=0x7f0fcc00b040).stop 2023-10-12T22:35:12.391 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.388+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/418541858 shutdown_connections 2023-10-12T22:35:12.391 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.388+0000 7f0fe0e7e700 1 --2- 172.21.15.195:0/418541858 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0fc402ff60 0x7f0fc4032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:12.391 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.388+0000 7f0fe0e7e700 1 --2- 172.21.15.195:0/418541858 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0fdc0f51a0 0x7f0fdc1086c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:12.391 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.388+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/418541858 >> 172.21.15.195:0/418541858 conn(0x7f0fdc0f09a0 msgr2=0x7f0fdc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:12.391 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.388+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/418541858 shutdown_connections 2023-10-12T22:35:12.391 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:12.388+0000 7f0fe0e7e700 1 -- 172.21.15.195:0/418541858 wait complete. 2023-10-12T22:35:12.392 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:12.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:10.832524+0000 mgr.smithi131.efktma (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:13.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:13 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:12.387937+0000 mon.smithi131 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.195:0/418541858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:13.777 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:13.778 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:14.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:14 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:12.832856+0000 mgr.smithi131.efktma (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:14.936 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.934+0000 7f2098d54700 1 -- 172.21.15.195:0/3928900792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20940f5410 msgr2=0x7f20940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:14.936 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.934+0000 7f2098d54700 1 --2- 172.21.15.195:0/3928900792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20940f5410 0x7f20940f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f2084009fb0 tx=0x7f2084009b30).stop 2023-10-12T22:35:14.936 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.934+0000 7f2098d54700 1 -- 172.21.15.195:0/3928900792 shutdown_connections 2023-10-12T22:35:14.936 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.934+0000 7f2098d54700 1 --2- 172.21.15.195:0/3928900792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20940f5410 0x7f20940f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:14.937 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.934+0000 7f2098d54700 1 -- 172.21.15.195:0/3928900792 >> 172.21.15.195:0/3928900792 conn(0x7f20940f0b70 msgr2=0x7f20940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:14.937 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.934+0000 7f2098d54700 1 -- 172.21.15.195:0/3928900792 shutdown_connections 2023-10-12T22:35:14.937 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.934+0000 7f2098d54700 1 -- 172.21.15.195:0/3928900792 wait complete. 2023-10-12T22:35:14.939 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.935+0000 7f2098d54700 1 Processor -- start 2023-10-12T22:35:14.939 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.935+0000 7f2098d54700 1 -- start start 2023-10-12T22:35:14.939 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.935+0000 7f2098d54700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20940f5410 0x7f2094105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:14.939 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.935+0000 7f2098d54700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2094105b20 con 0x7f20940f5410 2023-10-12T22:35:14.940 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.935+0000 7f209259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20940f5410 0x7f2094105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:14.940 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.935+0000 7f209259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20940f5410 0x7f2094105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51672/0 (socket says 172.21.15.195:51672) 2023-10-12T22:35:14.940 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.935+0000 7f209259c700 1 -- 172.21.15.195:0/3317143396 learned_addr learned my addr 172.21.15.195:0/3317143396 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:14.940 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.936+0000 7f209259c700 1 -- 172.21.15.195:0/3317143396 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084005040 con 0x7f20940f5410 2023-10-12T22:35:14.940 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.936+0000 7f209259c700 1 --2- 172.21.15.195:0/3317143396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20940f5410 0x7f2094105630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f2084013510 tx=0x7f208400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:14.941 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.936+0000 7f208affd700 1 -- 172.21.15.195:0/3317143396 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2084009320 con 0x7f20940f5410 2023-10-12T22:35:14.941 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.936+0000 7f2098d54700 1 -- 172.21.15.195:0/3317143396 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2094105d20 con 0x7f20940f5410 2023-10-12T22:35:14.941 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.936+0000 7f208affd700 1 -- 172.21.15.195:0/3317143396 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2084013f00 con 0x7f20940f5410 2023-10-12T22:35:14.941 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.936+0000 7f208affd700 1 -- 172.21.15.195:0/3317143396 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2084014d90 con 0x7f20940f5410 2023-10-12T22:35:14.941 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.936+0000 7f2098d54700 1 -- 172.21.15.195:0/3317143396 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20941061b0 con 0x7f20940f5410 2023-10-12T22:35:14.942 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.937+0000 7f208affd700 1 -- 172.21.15.195:0/3317143396 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20840263a0 con 0x7f20940f5410 2023-10-12T22:35:14.942 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.937+0000 7f208affd700 1 --2- 172.21.15.195:0/3317143396 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f208002ff60 0x7f2080032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:14.942 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.937+0000 7f208affd700 1 -- 172.21.15.195:0/3317143396 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2084041000 con 0x7f20940f5410 2023-10-12T22:35:14.942 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.937+0000 7f2098d54700 1 -- 172.21.15.195:0/3317143396 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2094045ad0 con 0x7f20940f5410 2023-10-12T22:35:14.942 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.937+0000 7f2091d9b700 1 --2- 172.21.15.195:0/3317143396 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f208002ff60 0x7f2080032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:14.943 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.938+0000 7f2091d9b700 1 --2- 172.21.15.195:0/3317143396 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f208002ff60 0x7f2080032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f207c0096a0 tx=0x7f207c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:14.943 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:14.942+0000 7f208affd700 1 -- 172.21.15.195:0/3317143396 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2084012920 con 0x7f20940f5410 2023-10-12T22:35:15.153 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:15.152+0000 7f2098d54700 1 -- 172.21.15.195:0/3317143396 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20940faf50 con 0x7f20940f5410 2023-10-12T22:35:15.154 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:15.154+0000 7f208affd700 1 -- 172.21.15.195:0/3317143396 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f208402b020 con 0x7f20940f5410 2023-10-12T22:35:15.154 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:15.155 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:15.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:15.154+0000 7f2098d54700 1 -- 172.21.15.195:0/3317143396 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f208002ff60 msgr2=0x7f2080032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:15.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:15.154+0000 7f2098d54700 1 --2- 172.21.15.195:0/3317143396 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f208002ff60 0x7f2080032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f207c0096a0 tx=0x7f207c006b40).stop 2023-10-12T22:35:15.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:15.154+0000 7f2098d54700 1 -- 172.21.15.195:0/3317143396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20940f5410 msgr2=0x7f2094105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:15.156 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:15.154+0000 7f2098d54700 1 --2- 172.21.15.195:0/3317143396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20940f5410 0x7f2094105630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f2084013510 tx=0x7f208400b040).stop 2023-10-12T22:35:15.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:15.155+0000 7f2098d54700 1 -- 172.21.15.195:0/3317143396 shutdown_connections 2023-10-12T22:35:15.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:15.155+0000 7f2098d54700 1 --2- 172.21.15.195:0/3317143396 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f208002ff60 0x7f2080032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:15.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:15.155+0000 7f2098d54700 1 --2- 172.21.15.195:0/3317143396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20940f5410 0x7f2094105630 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:15.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:15.155+0000 7f2098d54700 1 -- 172.21.15.195:0/3317143396 >> 172.21.15.195:0/3317143396 conn(0x7f20940f0b70 msgr2=0x7f20940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:15.157 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:15.155+0000 7f2098d54700 1 -- 172.21.15.195:0/3317143396 shutdown_connections 2023-10-12T22:35:15.158 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:15.155+0000 7f2098d54700 1 -- 172.21.15.195:0/3317143396 wait complete. 2023-10-12T22:35:15.158 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:16.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:15 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:14.833196+0000 mgr.smithi131.efktma (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:16.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:15 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:15.154100+0000 mon.smithi131 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.195:0/3317143396' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:16.502 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:16.503 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:17.646 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.643+0000 7f61dffff700 1 -- 172.21.15.195:0/3500317393 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61e00601e0 msgr2=0x7f61e00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:17.646 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.643+0000 7f61dffff700 1 --2- 172.21.15.195:0/3500317393 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61e00601e0 0x7f61e00f8f20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f61d0009fb0 tx=0x7f61d0009b30).stop 2023-10-12T22:35:17.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.643+0000 7f61dffff700 1 -- 172.21.15.195:0/3500317393 shutdown_connections 2023-10-12T22:35:17.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.643+0000 7f61dffff700 1 --2- 172.21.15.195:0/3500317393 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61e00601e0 0x7f61e00f8f20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:17.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.643+0000 7f61dffff700 1 -- 172.21.15.195:0/3500317393 >> 172.21.15.195:0/3500317393 conn(0x7f61e00f09a0 msgr2=0x7f61e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:17.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.643+0000 7f61dffff700 1 -- 172.21.15.195:0/3500317393 shutdown_connections 2023-10-12T22:35:17.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.643+0000 7f61dffff700 1 -- 172.21.15.195:0/3500317393 wait complete. 2023-10-12T22:35:17.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.643+0000 7f61dffff700 1 Processor -- start 2023-10-12T22:35:17.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.644+0000 7f61dffff700 1 -- start start 2023-10-12T22:35:17.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.644+0000 7f61dffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61e00601e0 0x7f61e0108720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:17.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.644+0000 7f61dffff700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e0108c10 con 0x7f61e00601e0 2023-10-12T22:35:17.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.644+0000 7f61deffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61e00601e0 0x7f61e0108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:17.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.644+0000 7f61deffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61e00601e0 0x7f61e0108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51686/0 (socket says 172.21.15.195:51686) 2023-10-12T22:35:17.649 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.644+0000 7f61deffd700 1 -- 172.21.15.195:0/929771503 learned_addr learned my addr 172.21.15.195:0/929771503 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:17.649 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.644+0000 7f61deffd700 1 -- 172.21.15.195:0/929771503 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d0005040 con 0x7f61e00601e0 2023-10-12T22:35:17.649 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.644+0000 7f61deffd700 1 --2- 172.21.15.195:0/929771503 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61e00601e0 0x7f61e0108720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f61d0013510 tx=0x7f61d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:17.649 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.645+0000 7f61c77fe700 1 -- 172.21.15.195:0/929771503 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61d0009320 con 0x7f61e00601e0 2023-10-12T22:35:17.649 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.645+0000 7f61dffff700 1 -- 172.21.15.195:0/929771503 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e0108e10 con 0x7f61e00601e0 2023-10-12T22:35:17.650 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.645+0000 7f61c77fe700 1 -- 172.21.15.195:0/929771503 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61d0013f00 con 0x7f61e00601e0 2023-10-12T22:35:17.650 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.645+0000 7f61c77fe700 1 -- 172.21.15.195:0/929771503 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61d0014d90 con 0x7f61e00601e0 2023-10-12T22:35:17.650 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.645+0000 7f61dffff700 1 -- 172.21.15.195:0/929771503 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e01092a0 con 0x7f61e00601e0 2023-10-12T22:35:17.652 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.645+0000 7f61c77fe700 1 -- 172.21.15.195:0/929771503 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61d00263a0 con 0x7f61e00601e0 2023-10-12T22:35:17.652 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.646+0000 7f61c77fe700 1 --2- 172.21.15.195:0/929771503 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f61c802ff60 0x7f61c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:17.652 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.645+0000 7f61dffff700 1 -- 172.21.15.195:0/929771503 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e0045ad0 con 0x7f61e00601e0 2023-10-12T22:35:17.652 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.646+0000 7f61c77fe700 1 -- 172.21.15.195:0/929771503 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61d0041300 con 0x7f61e00601e0 2023-10-12T22:35:17.653 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.646+0000 7f61de7fc700 1 --2- 172.21.15.195:0/929771503 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f61c802ff60 0x7f61c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:17.653 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.646+0000 7f61de7fc700 1 --2- 172.21.15.195:0/929771503 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f61c802ff60 0x7f61c8032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f61d40096a0 tx=0x7f61d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:17.653 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.650+0000 7f61c77fe700 1 -- 172.21.15.195:0/929771503 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61d001d3d0 con 0x7f61e00601e0 2023-10-12T22:35:17.858 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.857+0000 7f61dffff700 1 -- 172.21.15.195:0/929771503 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61e010a150 con 0x7f61e00601e0 2023-10-12T22:35:17.859 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.859+0000 7f61c77fe700 1 -- 172.21.15.195:0/929771503 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f61d000b0b0 con 0x7f61e00601e0 2023-10-12T22:35:17.859 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:17.860 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:17.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.860+0000 7f61dffff700 1 -- 172.21.15.195:0/929771503 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f61c802ff60 msgr2=0x7f61c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:17.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.860+0000 7f61dffff700 1 --2- 172.21.15.195:0/929771503 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f61c802ff60 0x7f61c8032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f61d40096a0 tx=0x7f61d4006b40).stop 2023-10-12T22:35:17.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.860+0000 7f61dffff700 1 -- 172.21.15.195:0/929771503 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61e00601e0 msgr2=0x7f61e0108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:17.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.860+0000 7f61dffff700 1 --2- 172.21.15.195:0/929771503 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61e00601e0 0x7f61e0108720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f61d0013510 tx=0x7f61d000b040).stop 2023-10-12T22:35:17.862 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.860+0000 7f61dffff700 1 -- 172.21.15.195:0/929771503 shutdown_connections 2023-10-12T22:35:17.862 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.860+0000 7f61dffff700 1 --2- 172.21.15.195:0/929771503 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f61c802ff60 0x7f61c8032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:17.862 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.860+0000 7f61dffff700 1 --2- 172.21.15.195:0/929771503 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61e00601e0 0x7f61e0108720 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:17.862 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.860+0000 7f61dffff700 1 -- 172.21.15.195:0/929771503 >> 172.21.15.195:0/929771503 conn(0x7f61e00f09a0 msgr2=0x7f61e00f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:17.862 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.860+0000 7f61dffff700 1 -- 172.21.15.195:0/929771503 shutdown_connections 2023-10-12T22:35:17.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:17.860+0000 7f61dffff700 1 -- 172.21.15.195:0/929771503 wait complete. 2023-10-12T22:35:17.863 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:18.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:17 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:16.833528+0000 mgr.smithi131.efktma (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:18.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:17 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:17.859366+0000 mon.smithi131 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.195:0/929771503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:19.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:19.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:20.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:19 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:18.833885+0000 mgr.smithi131.efktma (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:20.435 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.433+0000 7fbd0a66b700 1 -- 172.21.15.195:0/2769269506 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04060280 msgr2=0x7fbd040f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:20.435 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.433+0000 7fbd0a66b700 1 --2- 172.21.15.195:0/2769269506 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04060280 0x7fbd040f90f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fbcf4009fb0 tx=0x7fbcf4009b30).stop 2023-10-12T22:35:20.435 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.433+0000 7fbd0a66b700 1 -- 172.21.15.195:0/2769269506 shutdown_connections 2023-10-12T22:35:20.436 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.433+0000 7fbd0a66b700 1 --2- 172.21.15.195:0/2769269506 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04060280 0x7fbd040f90f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:20.436 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.433+0000 7fbd0a66b700 1 -- 172.21.15.195:0/2769269506 >> 172.21.15.195:0/2769269506 conn(0x7fbd040f0b70 msgr2=0x7fbd040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:20.436 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.433+0000 7fbd0a66b700 1 -- 172.21.15.195:0/2769269506 shutdown_connections 2023-10-12T22:35:20.436 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.433+0000 7fbd0a66b700 1 -- 172.21.15.195:0/2769269506 wait complete. 2023-10-12T22:35:20.436 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.434+0000 7fbd0a66b700 1 Processor -- start 2023-10-12T22:35:20.437 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.434+0000 7fbd0a66b700 1 -- start start 2023-10-12T22:35:20.437 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.434+0000 7fbd0a66b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04060280 0x7fbd040f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:20.437 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.434+0000 7fbd0a66b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd040f4ff0 con 0x7fbd04060280 2023-10-12T22:35:20.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.434+0000 7fbd03fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04060280 0x7fbd040f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:20.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.434+0000 7fbd03fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04060280 0x7fbd040f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:36622/0 (socket says 172.21.15.195:36622) 2023-10-12T22:35:20.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.434+0000 7fbd03fff700 1 -- 172.21.15.195:0/30820606 learned_addr learned my addr 172.21.15.195:0/30820606 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:20.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.435+0000 7fbd03fff700 1 -- 172.21.15.195:0/30820606 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf4005040 con 0x7fbd04060280 2023-10-12T22:35:20.439 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.435+0000 7fbd03fff700 1 --2- 172.21.15.195:0/30820606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04060280 0x7fbd040f6b70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fbcf4013510 tx=0x7fbcf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:20.440 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.435+0000 7fbd00ff9700 1 -- 172.21.15.195:0/30820606 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcf4009320 con 0x7fbd04060280 2023-10-12T22:35:20.440 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.435+0000 7fbd0a66b700 1 -- 172.21.15.195:0/30820606 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd040f5190 con 0x7fbd04060280 2023-10-12T22:35:20.440 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.435+0000 7fbd00ff9700 1 -- 172.21.15.195:0/30820606 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcf4013f00 con 0x7fbd04060280 2023-10-12T22:35:20.440 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.435+0000 7fbd00ff9700 1 -- 172.21.15.195:0/30820606 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcf4014dc0 con 0x7fbd04060280 2023-10-12T22:35:20.440 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.435+0000 7fbd0a66b700 1 -- 172.21.15.195:0/30820606 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd040f5620 con 0x7fbd04060280 2023-10-12T22:35:20.441 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.436+0000 7fbd00ff9700 1 -- 172.21.15.195:0/30820606 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcf401d3d0 con 0x7fbd04060280 2023-10-12T22:35:20.441 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.436+0000 7fbd0a66b700 1 -- 172.21.15.195:0/30820606 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd04059100 con 0x7fbd04060280 2023-10-12T22:35:20.441 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.436+0000 7fbd00ff9700 1 --2- 172.21.15.195:0/30820606 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbcec02ff60 0x7fbcec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:20.441 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.436+0000 7fbd00ff9700 1 -- 172.21.15.195:0/30820606 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcf40413e0 con 0x7fbd04060280 2023-10-12T22:35:20.441 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.436+0000 7fbd037fe700 1 --2- 172.21.15.195:0/30820606 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbcec02ff60 0x7fbcec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:20.442 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.436+0000 7fbd037fe700 1 --2- 172.21.15.195:0/30820606 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbcec02ff60 0x7fbcec032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fbcf80096a0 tx=0x7fbcf8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:20.442 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.441+0000 7fbd00ff9700 1 -- 172.21.15.195:0/30820606 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcf401de40 con 0x7fbd04060280 2023-10-12T22:35:20.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.645+0000 7fbd0a66b700 1 -- 172.21.15.195:0/30820606 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd04045ad0 con 0x7fbd04060280 2023-10-12T22:35:20.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.646+0000 7fbd00ff9700 1 -- 172.21.15.195:0/30820606 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbcf4012330 con 0x7fbd04060280 2023-10-12T22:35:20.648 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:20.648 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:20.649 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.648+0000 7fbd0a66b700 1 -- 172.21.15.195:0/30820606 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbcec02ff60 msgr2=0x7fbcec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:20.649 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.648+0000 7fbd0a66b700 1 --2- 172.21.15.195:0/30820606 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbcec02ff60 0x7fbcec032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fbcf80096a0 tx=0x7fbcf8006b40).stop 2023-10-12T22:35:20.650 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.648+0000 7fbd0a66b700 1 -- 172.21.15.195:0/30820606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04060280 msgr2=0x7fbd040f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:20.650 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.648+0000 7fbd0a66b700 1 --2- 172.21.15.195:0/30820606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04060280 0x7fbd040f6b70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fbcf4013510 tx=0x7fbcf400b040).stop 2023-10-12T22:35:20.650 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.648+0000 7fbd0a66b700 1 -- 172.21.15.195:0/30820606 shutdown_connections 2023-10-12T22:35:20.650 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.648+0000 7fbd0a66b700 1 --2- 172.21.15.195:0/30820606 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbcec02ff60 0x7fbcec032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:20.651 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.648+0000 7fbd0a66b700 1 --2- 172.21.15.195:0/30820606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04060280 0x7fbd040f6b70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:20.651 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.648+0000 7fbd0a66b700 1 -- 172.21.15.195:0/30820606 >> 172.21.15.195:0/30820606 conn(0x7fbd040f0b70 msgr2=0x7fbd040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:20.651 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.648+0000 7fbd0a66b700 1 -- 172.21.15.195:0/30820606 shutdown_connections 2023-10-12T22:35:20.651 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:20.648+0000 7fbd0a66b700 1 -- 172.21.15.195:0/30820606 wait complete. 2023-10-12T22:35:20.652 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:21.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:20 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:20.647578+0000 mon.smithi131 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.195:0/30820606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:22.160 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:22.160 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:22.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:21 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:20.834265+0000 mgr.smithi131.efktma (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:23.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:22.933869+0000 mon.smithi131 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/mirror_snapshot_schedule"}]: dispatch 2023-10-12T22:35:23.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:22.940729+0000 mon.smithi131 (mon.0) 199 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/trash_purge_schedule"}]: dispatch 2023-10-12T22:35:23.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.320+0000 7fa440253700 1 -- 172.21.15.195:0/2283599629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa4380f5410 msgr2=0x7fa4380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:23.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.320+0000 7fa440253700 1 --2- 172.21.15.195:0/2283599629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa4380f5410 0x7fa4380f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fa428009fb0 tx=0x7fa428009b30).stop 2023-10-12T22:35:23.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.320+0000 7fa440253700 1 -- 172.21.15.195:0/2283599629 shutdown_connections 2023-10-12T22:35:23.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.320+0000 7fa440253700 1 --2- 172.21.15.195:0/2283599629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa4380f5410 0x7fa4380f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:23.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.320+0000 7fa440253700 1 -- 172.21.15.195:0/2283599629 >> 172.21.15.195:0/2283599629 conn(0x7fa4380f0b70 msgr2=0x7fa4380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:23.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.320+0000 7fa440253700 1 -- 172.21.15.195:0/2283599629 shutdown_connections 2023-10-12T22:35:23.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.320+0000 7fa440253700 1 -- 172.21.15.195:0/2283599629 wait complete. 2023-10-12T22:35:23.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.321+0000 7fa440253700 1 Processor -- start 2023-10-12T22:35:23.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.321+0000 7fa440253700 1 -- start start 2023-10-12T22:35:23.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.321+0000 7fa440253700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa4380f5410 0x7fa438106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:23.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.321+0000 7fa440253700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa438106c10 con 0x7fa4380f5410 2023-10-12T22:35:23.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.321+0000 7fa43dfef700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa4380f5410 0x7fa438106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:23.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.321+0000 7fa43dfef700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa4380f5410 0x7fa438106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:36644/0 (socket says 172.21.15.195:36644) 2023-10-12T22:35:23.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.322+0000 7fa43dfef700 1 -- 172.21.15.195:0/4083164018 learned_addr learned my addr 172.21.15.195:0/4083164018 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:23.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.322+0000 7fa43dfef700 1 -- 172.21.15.195:0/4083164018 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa428005040 con 0x7fa4380f5410 2023-10-12T22:35:23.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.322+0000 7fa43dfef700 1 --2- 172.21.15.195:0/4083164018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa4380f5410 0x7fa438106720 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fa428013510 tx=0x7fa42800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:23.326 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.322+0000 7fa42e7fc700 1 -- 172.21.15.195:0/4083164018 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa428009320 con 0x7fa4380f5410 2023-10-12T22:35:23.326 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.322+0000 7fa42e7fc700 1 -- 172.21.15.195:0/4083164018 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa428013f00 con 0x7fa4380f5410 2023-10-12T22:35:23.326 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.322+0000 7fa440253700 1 -- 172.21.15.195:0/4083164018 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa438106e10 con 0x7fa4380f5410 2023-10-12T22:35:23.326 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.322+0000 7fa42e7fc700 1 -- 172.21.15.195:0/4083164018 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa428014d90 con 0x7fa4380f5410 2023-10-12T22:35:23.326 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.322+0000 7fa440253700 1 -- 172.21.15.195:0/4083164018 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4381072a0 con 0x7fa4380f5410 2023-10-12T22:35:23.327 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.323+0000 7fa42e7fc700 1 -- 172.21.15.195:0/4083164018 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa428014420 con 0x7fa4380f5410 2023-10-12T22:35:23.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.323+0000 7fa440253700 1 -- 172.21.15.195:0/4083164018 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa438045ad0 con 0x7fa4380f5410 2023-10-12T22:35:23.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.323+0000 7fa42e7fc700 1 --2- 172.21.15.195:0/4083164018 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa42402ff60 0x7fa424032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:23.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.323+0000 7fa42e7fc700 1 -- 172.21.15.195:0/4083164018 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa428041080 con 0x7fa4380f5410 2023-10-12T22:35:23.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.323+0000 7fa43d7ee700 1 --2- 172.21.15.195:0/4083164018 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa42402ff60 0x7fa424032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:23.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.324+0000 7fa43d7ee700 1 --2- 172.21.15.195:0/4083164018 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa42402ff60 0x7fa424032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa4340096a0 tx=0x7fa434006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:23.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.328+0000 7fa42e7fc700 1 -- 172.21.15.195:0/4083164018 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa42801dd50 con 0x7fa4380f5410 2023-10-12T22:35:23.534 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.534+0000 7fa440253700 1 -- 172.21.15.195:0/4083164018 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa438108150 con 0x7fa4380f5410 2023-10-12T22:35:23.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.534+0000 7fa42e7fc700 1 -- 172.21.15.195:0/4083164018 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa42801db70 con 0x7fa4380f5410 2023-10-12T22:35:23.535 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:23.535 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:23.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.536+0000 7fa440253700 1 -- 172.21.15.195:0/4083164018 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa42402ff60 msgr2=0x7fa424032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:23.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.536+0000 7fa440253700 1 --2- 172.21.15.195:0/4083164018 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa42402ff60 0x7fa424032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa4340096a0 tx=0x7fa434006b40).stop 2023-10-12T22:35:23.537 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.536+0000 7fa440253700 1 -- 172.21.15.195:0/4083164018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa4380f5410 msgr2=0x7fa438106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:23.537 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.536+0000 7fa440253700 1 --2- 172.21.15.195:0/4083164018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa4380f5410 0x7fa438106720 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fa428013510 tx=0x7fa42800b040).stop 2023-10-12T22:35:23.537 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.536+0000 7fa440253700 1 -- 172.21.15.195:0/4083164018 shutdown_connections 2023-10-12T22:35:23.537 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.536+0000 7fa440253700 1 --2- 172.21.15.195:0/4083164018 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa42402ff60 0x7fa424032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:23.537 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.536+0000 7fa440253700 1 --2- 172.21.15.195:0/4083164018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa4380f5410 0x7fa438106720 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:23.538 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.536+0000 7fa440253700 1 -- 172.21.15.195:0/4083164018 >> 172.21.15.195:0/4083164018 conn(0x7fa4380f0b70 msgr2=0x7fa43805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:23.538 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.536+0000 7fa440253700 1 -- 172.21.15.195:0/4083164018 shutdown_connections 2023-10-12T22:35:23.538 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:23.536+0000 7fa440253700 1 -- 172.21.15.195:0/4083164018 wait complete. 2023-10-12T22:35:23.538 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:24.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:22.834573+0000 mgr.smithi131.efktma (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:24.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:23.534816+0000 mon.smithi131 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.195:0/4083164018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:24.909 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:24.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:26.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.040+0000 7f0050f19700 1 -- 172.21.15.195:0/1129760721 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f004c060200 msgr2=0x7f004c0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:26.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.040+0000 7f0050f19700 1 --2- 172.21.15.195:0/1129760721 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f004c060200 0x7f004c0fb570 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f0034009fb0 tx=0x7f0034009b30).stop 2023-10-12T22:35:26.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.040+0000 7f0050f19700 1 -- 172.21.15.195:0/1129760721 shutdown_connections 2023-10-12T22:35:26.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.040+0000 7f0050f19700 1 --2- 172.21.15.195:0/1129760721 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f004c060200 0x7f004c0fb570 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:26.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.040+0000 7f0050f19700 1 -- 172.21.15.195:0/1129760721 >> 172.21.15.195:0/1129760721 conn(0x7f004c0f0b70 msgr2=0x7f004c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:26.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.040+0000 7f0050f19700 1 -- 172.21.15.195:0/1129760721 shutdown_connections 2023-10-12T22:35:26.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.040+0000 7f0050f19700 1 -- 172.21.15.195:0/1129760721 wait complete. 2023-10-12T22:35:26.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.041+0000 7f0050f19700 1 Processor -- start 2023-10-12T22:35:26.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.041+0000 7f0050f19700 1 -- start start 2023-10-12T22:35:26.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.041+0000 7f0050f19700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f004c060200 0x7f004c1088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:26.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.041+0000 7f0050f19700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004c108da0 con 0x7f004c060200 2023-10-12T22:35:26.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.041+0000 7f004a59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f004c060200 0x7f004c1088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:26.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.041+0000 7f004a59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f004c060200 0x7f004c1088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:36652/0 (socket says 172.21.15.195:36652) 2023-10-12T22:35:26.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.041+0000 7f004a59c700 1 -- 172.21.15.195:0/2278192630 learned_addr learned my addr 172.21.15.195:0/2278192630 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:26.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.042+0000 7f004a59c700 1 -- 172.21.15.195:0/2278192630 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0034005040 con 0x7f004c060200 2023-10-12T22:35:26.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.042+0000 7f004a59c700 1 --2- 172.21.15.195:0/2278192630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f004c060200 0x7f004c1088b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f0034013510 tx=0x7f003400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:26.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.042+0000 7f0042ffd700 1 -- 172.21.15.195:0/2278192630 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0034009320 con 0x7f004c060200 2023-10-12T22:35:26.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.042+0000 7f0050f19700 1 -- 172.21.15.195:0/2278192630 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f004c108fa0 con 0x7f004c060200 2023-10-12T22:35:26.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.042+0000 7f0042ffd700 1 -- 172.21.15.195:0/2278192630 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0034013f00 con 0x7f004c060200 2023-10-12T22:35:26.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.042+0000 7f0042ffd700 1 -- 172.21.15.195:0/2278192630 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0034014dc0 con 0x7f004c060200 2023-10-12T22:35:26.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.042+0000 7f0050f19700 1 -- 172.21.15.195:0/2278192630 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f004c109430 con 0x7f004c060200 2023-10-12T22:35:26.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.043+0000 7f0042ffd700 1 -- 172.21.15.195:0/2278192630 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0034014420 con 0x7f004c060200 2023-10-12T22:35:26.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.043+0000 7f0050f19700 1 -- 172.21.15.195:0/2278192630 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f004c045ad0 con 0x7f004c060200 2023-10-12T22:35:26.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.043+0000 7f0042ffd700 1 --2- 172.21.15.195:0/2278192630 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f003802ff60 0x7f0038032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:26.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.043+0000 7f0042ffd700 1 -- 172.21.15.195:0/2278192630 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0034041110 con 0x7f004c060200 2023-10-12T22:35:26.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.043+0000 7f0049d9b700 1 --2- 172.21.15.195:0/2278192630 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f003802ff60 0x7f0038032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:26.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.044+0000 7f0049d9b700 1 --2- 172.21.15.195:0/2278192630 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f003802ff60 0x7f0038032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f003c0096a0 tx=0x7f003c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:26.050 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.048+0000 7f0042ffd700 1 -- 172.21.15.195:0/2278192630 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f00340095f0 con 0x7f004c060200 2023-10-12T22:35:26.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:25 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:24.834913+0000 mgr.smithi131.efktma (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:26.255 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.255+0000 7f0050f19700 1 -- 172.21.15.195:0/2278192630 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f004c10a2e0 con 0x7f004c060200 2023-10-12T22:35:26.256 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.256+0000 7f0042ffd700 1 -- 172.21.15.195:0/2278192630 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f003401cb10 con 0x7f004c060200 2023-10-12T22:35:26.256 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:26.257 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:26.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.258+0000 7f0050f19700 1 -- 172.21.15.195:0/2278192630 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f003802ff60 msgr2=0x7f0038032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:26.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.258+0000 7f0050f19700 1 --2- 172.21.15.195:0/2278192630 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f003802ff60 0x7f0038032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f003c0096a0 tx=0x7f003c006b40).stop 2023-10-12T22:35:26.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.258+0000 7f0050f19700 1 -- 172.21.15.195:0/2278192630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f004c060200 msgr2=0x7f004c1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:26.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.258+0000 7f0050f19700 1 --2- 172.21.15.195:0/2278192630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f004c060200 0x7f004c1088b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f0034013510 tx=0x7f003400b040).stop 2023-10-12T22:35:26.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.258+0000 7f0050f19700 1 -- 172.21.15.195:0/2278192630 shutdown_connections 2023-10-12T22:35:26.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.258+0000 7f0050f19700 1 --2- 172.21.15.195:0/2278192630 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f003802ff60 0x7f0038032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:26.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.258+0000 7f0050f19700 1 --2- 172.21.15.195:0/2278192630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f004c060200 0x7f004c1088b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:26.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.258+0000 7f0050f19700 1 -- 172.21.15.195:0/2278192630 >> 172.21.15.195:0/2278192630 conn(0x7f004c0f0b70 msgr2=0x7f004c0f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:26.259 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.258+0000 7f0050f19700 1 -- 172.21.15.195:0/2278192630 shutdown_connections 2023-10-12T22:35:26.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:26.258+0000 7f0050f19700 1 -- 172.21.15.195:0/2278192630 wait complete. 2023-10-12T22:35:26.260 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:27.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:26 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:26.256407+0000 mon.smithi131 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.195:0/2278192630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:27.637 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:27.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:28.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:27 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:26.835238+0000 mgr.smithi131.efktma (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:28.809 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.808+0000 7f110e306700 1 -- 172.21.15.195:0/3259569115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11080f5410 msgr2=0x7f11080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:28.809 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.808+0000 7f110e306700 1 --2- 172.21.15.195:0/3259569115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11080f5410 0x7f11080f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f10f0009fb0 tx=0x7f10f0009b30).stop 2023-10-12T22:35:28.809 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.808+0000 7f110e306700 1 -- 172.21.15.195:0/3259569115 shutdown_connections 2023-10-12T22:35:28.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.808+0000 7f110e306700 1 --2- 172.21.15.195:0/3259569115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11080f5410 0x7f11080f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:28.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.808+0000 7f110e306700 1 -- 172.21.15.195:0/3259569115 >> 172.21.15.195:0/3259569115 conn(0x7f11080f0b70 msgr2=0x7f11080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:28.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.808+0000 7f110e306700 1 -- 172.21.15.195:0/3259569115 shutdown_connections 2023-10-12T22:35:28.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.808+0000 7f110e306700 1 -- 172.21.15.195:0/3259569115 wait complete. 2023-10-12T22:35:28.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.808+0000 7f110e306700 1 Processor -- start 2023-10-12T22:35:28.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.809+0000 7f110e306700 1 -- start start 2023-10-12T22:35:28.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.809+0000 7f110e306700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11080f5410 0x7f1108109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:28.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.809+0000 7f110e306700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1108060240 con 0x7f11080f5410 2023-10-12T22:35:28.813 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.809+0000 7f1107fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11080f5410 0x7f1108109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:28.813 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.809+0000 7f1107fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11080f5410 0x7f1108109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38456/0 (socket says 172.21.15.195:38456) 2023-10-12T22:35:28.813 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.809+0000 7f1107fff700 1 -- 172.21.15.195:0/717545213 learned_addr learned my addr 172.21.15.195:0/717545213 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:28.813 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.809+0000 7f1107fff700 1 -- 172.21.15.195:0/717545213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f0005040 con 0x7f11080f5410 2023-10-12T22:35:28.813 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.809+0000 7f1107fff700 1 --2- 172.21.15.195:0/717545213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11080f5410 0x7f1108109430 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f10f0013510 tx=0x7f10f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:28.814 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.810+0000 7f1104ff9700 1 -- 172.21.15.195:0/717545213 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10f0009320 con 0x7f11080f5410 2023-10-12T22:35:28.814 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.810+0000 7f110e306700 1 -- 172.21.15.195:0/717545213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1108060440 con 0x7f11080f5410 2023-10-12T22:35:28.814 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.810+0000 7f1104ff9700 1 -- 172.21.15.195:0/717545213 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10f0013f00 con 0x7f11080f5410 2023-10-12T22:35:28.814 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.810+0000 7f1104ff9700 1 -- 172.21.15.195:0/717545213 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10f0014d90 con 0x7f11080f5410 2023-10-12T22:35:28.814 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.810+0000 7f110e306700 1 -- 172.21.15.195:0/717545213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1108109ae0 con 0x7f11080f5410 2023-10-12T22:35:28.815 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.811+0000 7f1104ff9700 1 -- 172.21.15.195:0/717545213 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10f00095f0 con 0x7f11080f5410 2023-10-12T22:35:28.815 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.811+0000 7f110e306700 1 -- 172.21.15.195:0/717545213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1108045ad0 con 0x7f11080f5410 2023-10-12T22:35:28.815 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.811+0000 7f1104ff9700 1 --2- 172.21.15.195:0/717545213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f10f402ff60 0x7f10f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:28.815 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.811+0000 7f1104ff9700 1 -- 172.21.15.195:0/717545213 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f10f0040ff0 con 0x7f11080f5410 2023-10-12T22:35:28.816 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.811+0000 7f11077fe700 1 --2- 172.21.15.195:0/717545213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f10f402ff60 0x7f10f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:28.816 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.812+0000 7f11077fe700 1 --2- 172.21.15.195:0/717545213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f10f402ff60 0x7f10f4032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f10f80096a0 tx=0x7f10f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:28.816 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:28.816+0000 7f1104ff9700 1 -- 172.21.15.195:0/717545213 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10f0014420 con 0x7f11080f5410 2023-10-12T22:35:29.020 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:29.020+0000 7f110e306700 1 -- 172.21.15.195:0/717545213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11080fae60 con 0x7f11080f5410 2023-10-12T22:35:29.021 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:29.021+0000 7f1104ff9700 1 -- 172.21.15.195:0/717545213 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f10f0010230 con 0x7f11080f5410 2023-10-12T22:35:29.021 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:29.022 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:29.023 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:29.022+0000 7f110e306700 1 -- 172.21.15.195:0/717545213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f10f402ff60 msgr2=0x7f10f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:29.023 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:29.022+0000 7f110e306700 1 --2- 172.21.15.195:0/717545213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f10f402ff60 0x7f10f4032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f10f80096a0 tx=0x7f10f8006b40).stop 2023-10-12T22:35:29.023 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:29.023+0000 7f110e306700 1 -- 172.21.15.195:0/717545213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11080f5410 msgr2=0x7f1108109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:29.023 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:29.023+0000 7f110e306700 1 --2- 172.21.15.195:0/717545213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11080f5410 0x7f1108109430 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f10f0013510 tx=0x7f10f000b040).stop 2023-10-12T22:35:29.024 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:29.023+0000 7f110e306700 1 -- 172.21.15.195:0/717545213 shutdown_connections 2023-10-12T22:35:29.024 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:29.023+0000 7f110e306700 1 --2- 172.21.15.195:0/717545213 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f10f402ff60 0x7f10f4032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:29.024 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:29.023+0000 7f110e306700 1 --2- 172.21.15.195:0/717545213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f11080f5410 0x7f1108109430 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:29.024 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:29.023+0000 7f110e306700 1 -- 172.21.15.195:0/717545213 >> 172.21.15.195:0/717545213 conn(0x7f11080f0b70 msgr2=0x7f11080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:29.024 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:29.023+0000 7f110e306700 1 -- 172.21.15.195:0/717545213 shutdown_connections 2023-10-12T22:35:29.024 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:29.023+0000 7f110e306700 1 -- 172.21.15.195:0/717545213 wait complete. 2023-10-12T22:35:29.025 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:30.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:28.835493+0000 mgr.smithi131.efktma (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:30.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:29.021279+0000 mon.smithi131 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.195:0/717545213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:30.393 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:30.393 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:31.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.557+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/1927229320 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fc40f5410 msgr2=0x7f7fc40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:31.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.557+0000 7f7fc8a5b700 1 --2- 172.21.15.195:0/1927229320 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fc40f5410 0x7f7fc40f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f7fac009fb0 tx=0x7f7fac009b30).stop 2023-10-12T22:35:31.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.557+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/1927229320 shutdown_connections 2023-10-12T22:35:31.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.557+0000 7f7fc8a5b700 1 --2- 172.21.15.195:0/1927229320 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fc40f5410 0x7f7fc40f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:31.560 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.557+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/1927229320 >> 172.21.15.195:0/1927229320 conn(0x7f7fc40f0b70 msgr2=0x7f7fc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:31.560 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.557+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/1927229320 shutdown_connections 2023-10-12T22:35:31.560 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.557+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/1927229320 wait complete. 2023-10-12T22:35:31.560 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.558+0000 7f7fc8a5b700 1 Processor -- start 2023-10-12T22:35:31.560 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.558+0000 7f7fc8a5b700 1 -- start start 2023-10-12T22:35:31.561 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.558+0000 7f7fc8a5b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fc40f5410 0x7f7fc4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:31.561 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.558+0000 7f7fc8a5b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc4105b20 con 0x7f7fc40f5410 2023-10-12T22:35:31.561 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.558+0000 7f7fc259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fc40f5410 0x7f7fc4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:31.561 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.558+0000 7f7fc259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fc40f5410 0x7f7fc4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38474/0 (socket says 172.21.15.195:38474) 2023-10-12T22:35:31.561 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.558+0000 7f7fc259c700 1 -- 172.21.15.195:0/2084468430 learned_addr learned my addr 172.21.15.195:0/2084468430 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:31.561 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.559+0000 7f7fc259c700 1 -- 172.21.15.195:0/2084468430 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac005040 con 0x7f7fc40f5410 2023-10-12T22:35:31.562 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.559+0000 7f7fc259c700 1 --2- 172.21.15.195:0/2084468430 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fc40f5410 0x7f7fc4105630 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f7fac013510 tx=0x7f7fac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:31.563 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.559+0000 7f7fbaffd700 1 -- 172.21.15.195:0/2084468430 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fac009320 con 0x7f7fc40f5410 2023-10-12T22:35:31.563 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.559+0000 7f7fbaffd700 1 -- 172.21.15.195:0/2084468430 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fac013f00 con 0x7f7fc40f5410 2023-10-12T22:35:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.559+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/2084468430 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc4105d20 con 0x7f7fc40f5410 2023-10-12T22:35:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.559+0000 7f7fbaffd700 1 -- 172.21.15.195:0/2084468430 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fac014d90 con 0x7f7fc40f5410 2023-10-12T22:35:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.559+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/2084468430 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc41061b0 con 0x7f7fc40f5410 2023-10-12T22:35:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.560+0000 7f7fbaffd700 1 -- 172.21.15.195:0/2084468430 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7fac0095f0 con 0x7f7fc40f5410 2023-10-12T22:35:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.560+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/2084468430 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc4045ad0 con 0x7f7fc40f5410 2023-10-12T22:35:31.565 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.560+0000 7f7fbaffd700 1 --2- 172.21.15.195:0/2084468430 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7fb002ff60 0x7f7fb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:31.565 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.560+0000 7f7fbaffd700 1 -- 172.21.15.195:0/2084468430 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7fac0410c0 con 0x7f7fc40f5410 2023-10-12T22:35:31.565 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.560+0000 7f7fc1d9b700 1 --2- 172.21.15.195:0/2084468430 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7fb002ff60 0x7f7fb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:31.565 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.561+0000 7f7fc1d9b700 1 --2- 172.21.15.195:0/2084468430 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7fb002ff60 0x7f7fb0032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7fb40096a0 tx=0x7f7fb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:31.566 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.565+0000 7f7fbaffd700 1 -- 172.21.15.195:0/2084468430 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fac01d7f0 con 0x7f7fc40f5410 2023-10-12T22:35:31.774 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.774+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/2084468430 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fc40fae60 con 0x7f7fc40f5410 2023-10-12T22:35:31.774 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.774+0000 7f7fbaffd700 1 -- 172.21.15.195:0/2084468430 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7fac010230 con 0x7f7fc40f5410 2023-10-12T22:35:31.775 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:31.775 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:31.776 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.776+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/2084468430 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7fb002ff60 msgr2=0x7f7fb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:31.776 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.776+0000 7f7fc8a5b700 1 --2- 172.21.15.195:0/2084468430 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7fb002ff60 0x7f7fb0032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7fb40096a0 tx=0x7f7fb4006b40).stop 2023-10-12T22:35:31.776 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.776+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/2084468430 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fc40f5410 msgr2=0x7f7fc4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:31.777 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.776+0000 7f7fc8a5b700 1 --2- 172.21.15.195:0/2084468430 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fc40f5410 0x7f7fc4105630 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f7fac013510 tx=0x7f7fac00b040).stop 2023-10-12T22:35:31.777 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.776+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/2084468430 shutdown_connections 2023-10-12T22:35:31.777 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.776+0000 7f7fc8a5b700 1 --2- 172.21.15.195:0/2084468430 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7fb002ff60 0x7f7fb0032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:31.777 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.776+0000 7f7fc8a5b700 1 --2- 172.21.15.195:0/2084468430 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fc40f5410 0x7f7fc4105630 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:31.777 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.776+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/2084468430 >> 172.21.15.195:0/2084468430 conn(0x7f7fc40f0b70 msgr2=0x7f7fc40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:31.777 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.776+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/2084468430 shutdown_connections 2023-10-12T22:35:31.778 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:31.776+0000 7f7fc8a5b700 1 -- 172.21.15.195:0/2084468430 wait complete. 2023-10-12T22:35:31.778 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:32.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:31 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:30.835768+0000 mgr.smithi131.efktma (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:32.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:31 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:31.774611+0000 mon.smithi131 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.195:0/2084468430' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:33.152 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:33.153 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:34.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.215+0000 7fc918bbf700 1 -- 172.21.15.195:0/3404419762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9140f51a0 msgr2=0x7fc9140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:34.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.215+0000 7fc918bbf700 1 --2- 172.21.15.195:0/3404419762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9140f51a0 0x7fc9140f55c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fc904009fb0 tx=0x7fc904009b30).stop 2023-10-12T22:35:34.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.215+0000 7fc918bbf700 1 -- 172.21.15.195:0/3404419762 shutdown_connections 2023-10-12T22:35:34.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.215+0000 7fc918bbf700 1 --2- 172.21.15.195:0/3404419762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9140f51a0 0x7fc9140f55c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:34.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.215+0000 7fc918bbf700 1 -- 172.21.15.195:0/3404419762 >> 172.21.15.195:0/3404419762 conn(0x7fc9140f09a0 msgr2=0x7fc9140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:34.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.215+0000 7fc918bbf700 1 -- 172.21.15.195:0/3404419762 shutdown_connections 2023-10-12T22:35:34.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.215+0000 7fc918bbf700 1 -- 172.21.15.195:0/3404419762 wait complete. 2023-10-12T22:35:34.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.215+0000 7fc918bbf700 1 Processor -- start 2023-10-12T22:35:34.218 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.215+0000 7fc918bbf700 1 -- start start 2023-10-12T22:35:34.218 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.216+0000 7fc918bbf700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9140f51a0 0x7fc9141091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:34.218 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.216+0000 7fc918bbf700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc914060220 con 0x7fc9140f51a0 2023-10-12T22:35:34.218 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.216+0000 7fc9137fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9140f51a0 0x7fc9141091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:34.219 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.216+0000 7fc9137fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9140f51a0 0x7fc9141091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38486/0 (socket says 172.21.15.195:38486) 2023-10-12T22:35:34.219 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.216+0000 7fc9137fe700 1 -- 172.21.15.195:0/2380068509 learned_addr learned my addr 172.21.15.195:0/2380068509 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:34.219 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.216+0000 7fc9137fe700 1 -- 172.21.15.195:0/2380068509 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc904005040 con 0x7fc9140f51a0 2023-10-12T22:35:34.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.216+0000 7fc9137fe700 1 --2- 172.21.15.195:0/2380068509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9140f51a0 0x7fc9141091d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc904009830 tx=0x7fc90400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:34.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.216+0000 7fc8fbfff700 1 -- 172.21.15.195:0/2380068509 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc904009320 con 0x7fc9140f51a0 2023-10-12T22:35:34.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.216+0000 7fc918bbf700 1 -- 172.21.15.195:0/2380068509 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc914060420 con 0x7fc9140f51a0 2023-10-12T22:35:34.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.216+0000 7fc8fbfff700 1 -- 172.21.15.195:0/2380068509 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9040136c0 con 0x7fc9140f51a0 2023-10-12T22:35:34.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.217+0000 7fc8fbfff700 1 -- 172.21.15.195:0/2380068509 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc90401ee90 con 0x7fc9140f51a0 2023-10-12T22:35:34.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.217+0000 7fc918bbf700 1 -- 172.21.15.195:0/2380068509 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc914109880 con 0x7fc9140f51a0 2023-10-12T22:35:34.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.217+0000 7fc8fbfff700 1 -- 172.21.15.195:0/2380068509 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc90401e420 con 0x7fc9140f51a0 2023-10-12T22:35:34.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.217+0000 7fc918bbf700 1 -- 172.21.15.195:0/2380068509 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc914059100 con 0x7fc9140f51a0 2023-10-12T22:35:34.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.218+0000 7fc8fbfff700 1 --2- 172.21.15.195:0/2380068509 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc8fc02ff10 0x7fc8fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:34.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.218+0000 7fc8fbfff700 1 -- 172.21.15.195:0/2380068509 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc904040f40 con 0x7fc9140f51a0 2023-10-12T22:35:34.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.218+0000 7fc912ffd700 1 --2- 172.21.15.195:0/2380068509 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc8fc02ff10 0x7fc8fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:34.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.218+0000 7fc912ffd700 1 --2- 172.21.15.195:0/2380068509 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc8fc02ff10 0x7fc8fc0323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fc9080096a0 tx=0x7fc908006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:34.223 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.222+0000 7fc8fbfff700 1 -- 172.21.15.195:0/2380068509 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc90406d2b0 con 0x7fc9140f51a0 2023-10-12T22:35:34.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:33 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:32.836046+0000 mgr.smithi131.efktma (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:34.434 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.434+0000 7fc918bbf700 1 -- 172.21.15.195:0/2380068509 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9140fabe0 con 0x7fc9140f51a0 2023-10-12T22:35:34.435 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.434+0000 7fc8fbfff700 1 -- 172.21.15.195:0/2380068509 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc9040095f0 con 0x7fc9140f51a0 2023-10-12T22:35:34.435 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:34.435 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:34.436 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.436+0000 7fc918bbf700 1 -- 172.21.15.195:0/2380068509 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc8fc02ff10 msgr2=0x7fc8fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:34.437 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.436+0000 7fc918bbf700 1 --2- 172.21.15.195:0/2380068509 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc8fc02ff10 0x7fc8fc0323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fc9080096a0 tx=0x7fc908006b40).stop 2023-10-12T22:35:34.437 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.436+0000 7fc918bbf700 1 -- 172.21.15.195:0/2380068509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9140f51a0 msgr2=0x7fc9141091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:34.437 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.436+0000 7fc918bbf700 1 --2- 172.21.15.195:0/2380068509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9140f51a0 0x7fc9141091d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc904009830 tx=0x7fc90400b040).stop 2023-10-12T22:35:34.437 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.436+0000 7fc918bbf700 1 -- 172.21.15.195:0/2380068509 shutdown_connections 2023-10-12T22:35:34.437 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.436+0000 7fc918bbf700 1 --2- 172.21.15.195:0/2380068509 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc8fc02ff10 0x7fc8fc0323d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:34.438 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.436+0000 7fc918bbf700 1 --2- 172.21.15.195:0/2380068509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9140f51a0 0x7fc9141091d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:34.438 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.436+0000 7fc918bbf700 1 -- 172.21.15.195:0/2380068509 >> 172.21.15.195:0/2380068509 conn(0x7fc9140f09a0 msgr2=0x7fc9140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:34.438 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.436+0000 7fc918bbf700 1 -- 172.21.15.195:0/2380068509 shutdown_connections 2023-10-12T22:35:34.438 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:34.436+0000 7fc918bbf700 1 -- 172.21.15.195:0/2380068509 wait complete. 2023-10-12T22:35:34.439 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:35.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:34.434709+0000 mon.smithi131 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.195:0/2380068509' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:35.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:35.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:36.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:35 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:34.836328+0000 mgr.smithi131.efktma (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:36.949 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.947+0000 7f2647043700 1 -- 172.21.15.195:0/4250474809 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26400f53f0 msgr2=0x7f26400f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:36.949 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.947+0000 7f2647043700 1 --2- 172.21.15.195:0/4250474809 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26400f53f0 0x7f26400f5810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f2634009fb0 tx=0x7f2634009b30).stop 2023-10-12T22:35:36.949 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.947+0000 7f2647043700 1 -- 172.21.15.195:0/4250474809 shutdown_connections 2023-10-12T22:35:36.949 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.947+0000 7f2647043700 1 --2- 172.21.15.195:0/4250474809 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26400f53f0 0x7f26400f5810 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:36.950 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.947+0000 7f2647043700 1 -- 172.21.15.195:0/4250474809 >> 172.21.15.195:0/4250474809 conn(0x7f26400f0b70 msgr2=0x7f26400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:36.950 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.947+0000 7f2647043700 1 -- 172.21.15.195:0/4250474809 shutdown_connections 2023-10-12T22:35:36.950 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.947+0000 7f2647043700 1 -- 172.21.15.195:0/4250474809 wait complete. 2023-10-12T22:35:36.950 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.948+0000 7f2647043700 1 Processor -- start 2023-10-12T22:35:36.950 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.948+0000 7f2647043700 1 -- start start 2023-10-12T22:35:36.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.948+0000 7f2647043700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26400f53f0 0x7f26401093e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:36.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.948+0000 7f2647043700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2640060240 con 0x7f26400f53f0 2023-10-12T22:35:36.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.948+0000 7f2644ddf700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26400f53f0 0x7f26401093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:36.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.948+0000 7f2644ddf700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26400f53f0 0x7f26401093e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38506/0 (socket says 172.21.15.195:38506) 2023-10-12T22:35:36.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.948+0000 7f2644ddf700 1 -- 172.21.15.195:0/959135277 learned_addr learned my addr 172.21.15.195:0/959135277 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:36.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.949+0000 7f2644ddf700 1 -- 172.21.15.195:0/959135277 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2634005040 con 0x7f26400f53f0 2023-10-12T22:35:36.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.949+0000 7f2644ddf700 1 --2- 172.21.15.195:0/959135277 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26400f53f0 0x7f26401093e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f2634014000 tx=0x7f263400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:36.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.949+0000 7f263d7fa700 1 -- 172.21.15.195:0/959135277 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26340095f0 con 0x7f26400f53f0 2023-10-12T22:35:36.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.949+0000 7f2647043700 1 -- 172.21.15.195:0/959135277 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2640060440 con 0x7f26400f53f0 2023-10-12T22:35:36.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.949+0000 7f263d7fa700 1 -- 172.21.15.195:0/959135277 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2634007e70 con 0x7f26400f53f0 2023-10-12T22:35:36.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.949+0000 7f263d7fa700 1 -- 172.21.15.195:0/959135277 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2634014ea0 con 0x7f26400f53f0 2023-10-12T22:35:36.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.949+0000 7f2647043700 1 -- 172.21.15.195:0/959135277 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2640109a90 con 0x7f26400f53f0 2023-10-12T22:35:36.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.950+0000 7f263d7fa700 1 -- 172.21.15.195:0/959135277 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2634025070 con 0x7f26400f53f0 2023-10-12T22:35:36.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.950+0000 7f2647043700 1 -- 172.21.15.195:0/959135277 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2640045ad0 con 0x7f26400f53f0 2023-10-12T22:35:36.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.950+0000 7f263d7fa700 1 --2- 172.21.15.195:0/959135277 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f263002ff60 0x7f2630032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:36.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.950+0000 7f263d7fa700 1 -- 172.21.15.195:0/959135277 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26340409f0 con 0x7f26400f53f0 2023-10-12T22:35:36.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.950+0000 7f263ffff700 1 --2- 172.21.15.195:0/959135277 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f263002ff60 0x7f2630032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:36.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.951+0000 7f263ffff700 1 --2- 172.21.15.195:0/959135277 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f263002ff60 0x7f2630032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f262c0096a0 tx=0x7f262c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:36.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:36.955+0000 7f263d7fa700 1 -- 172.21.15.195:0/959135277 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f263400f4e0 con 0x7f26400f53f0 2023-10-12T22:35:37.160 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:37.159+0000 7f2647043700 1 -- 172.21.15.195:0/959135277 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f264010a2b0 con 0x7f26400f53f0 2023-10-12T22:35:37.161 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:37.160+0000 7f263d7fa700 1 -- 172.21.15.195:0/959135277 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f263400f300 con 0x7f26400f53f0 2023-10-12T22:35:37.161 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:37.161 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:37.163 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:37.161+0000 7f2647043700 1 -- 172.21.15.195:0/959135277 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f263002ff60 msgr2=0x7f2630032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:37.163 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:37.161+0000 7f2647043700 1 --2- 172.21.15.195:0/959135277 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f263002ff60 0x7f2630032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f262c0096a0 tx=0x7f262c006b40).stop 2023-10-12T22:35:37.163 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:37.161+0000 7f2647043700 1 -- 172.21.15.195:0/959135277 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26400f53f0 msgr2=0x7f26401093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:37.163 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:37.161+0000 7f2647043700 1 --2- 172.21.15.195:0/959135277 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26400f53f0 0x7f26401093e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f2634014000 tx=0x7f263400b040).stop 2023-10-12T22:35:37.164 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:37.162+0000 7f2647043700 1 -- 172.21.15.195:0/959135277 shutdown_connections 2023-10-12T22:35:37.164 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:37.162+0000 7f2647043700 1 --2- 172.21.15.195:0/959135277 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f263002ff60 0x7f2630032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:37.164 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:37.162+0000 7f2647043700 1 --2- 172.21.15.195:0/959135277 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26400f53f0 0x7f26401093e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:37.164 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:37.162+0000 7f2647043700 1 -- 172.21.15.195:0/959135277 >> 172.21.15.195:0/959135277 conn(0x7f26400f0b70 msgr2=0x7f26400f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:37.164 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:37.162+0000 7f2647043700 1 -- 172.21.15.195:0/959135277 shutdown_connections 2023-10-12T22:35:37.165 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:37.162+0000 7f2647043700 1 -- 172.21.15.195:0/959135277 wait complete. 2023-10-12T22:35:37.165 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:38.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:37 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:36.836666+0000 mgr.smithi131.efktma (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:38.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:37 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:37.161046+0000 mon.smithi131 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.195:0/959135277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:38.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:38.544 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:39.793 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.791+0000 7fcabca21700 1 -- 172.21.15.195:0/4046449825 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcab80601e0 msgr2=0x7fcab80f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:39.793 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.791+0000 7fcabca21700 1 --2- 172.21.15.195:0/4046449825 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcab80601e0 0x7fcab80f8f20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fcaa0009fb0 tx=0x7fcaa0009b30).stop 2023-10-12T22:35:39.793 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.791+0000 7fcabca21700 1 -- 172.21.15.195:0/4046449825 shutdown_connections 2023-10-12T22:35:39.793 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.791+0000 7fcabca21700 1 --2- 172.21.15.195:0/4046449825 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcab80601e0 0x7fcab80f8f20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:39.793 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.791+0000 7fcabca21700 1 -- 172.21.15.195:0/4046449825 >> 172.21.15.195:0/4046449825 conn(0x7fcab80f09a0 msgr2=0x7fcab80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:39.794 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.791+0000 7fcabca21700 1 -- 172.21.15.195:0/4046449825 shutdown_connections 2023-10-12T22:35:39.794 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.791+0000 7fcabca21700 1 -- 172.21.15.195:0/4046449825 wait complete. 2023-10-12T22:35:39.794 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.791+0000 7fcabca21700 1 Processor -- start 2023-10-12T22:35:39.796 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.792+0000 7fcabca21700 1 -- start start 2023-10-12T22:35:39.796 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.792+0000 7fcabca21700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcab80601e0 0x7fcab8108720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:39.796 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.792+0000 7fcabca21700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab8108c10 con 0x7fcab80601e0 2023-10-12T22:35:39.796 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.792+0000 7fcab77fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcab80601e0 0x7fcab8108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:39.797 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.792+0000 7fcab77fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcab80601e0 0x7fcab8108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:43248/0 (socket says 172.21.15.195:43248) 2023-10-12T22:35:39.797 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.792+0000 7fcab77fe700 1 -- 172.21.15.195:0/3217671072 learned_addr learned my addr 172.21.15.195:0/3217671072 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:39.797 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.792+0000 7fcab77fe700 1 -- 172.21.15.195:0/3217671072 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa0005040 con 0x7fcab80601e0 2023-10-12T22:35:39.797 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.793+0000 7fcab77fe700 1 --2- 172.21.15.195:0/3217671072 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcab80601e0 0x7fcab8108720 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fcaa0013510 tx=0x7fcaa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:39.797 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.793+0000 7fcaaffff700 1 -- 172.21.15.195:0/3217671072 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaa0009320 con 0x7fcab80601e0 2023-10-12T22:35:39.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.793+0000 7fcabca21700 1 -- 172.21.15.195:0/3217671072 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab8108e10 con 0x7fcab80601e0 2023-10-12T22:35:39.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.793+0000 7fcaaffff700 1 -- 172.21.15.195:0/3217671072 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcaa0013f00 con 0x7fcab80601e0 2023-10-12T22:35:39.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.793+0000 7fcaaffff700 1 -- 172.21.15.195:0/3217671072 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaa0014d90 con 0x7fcab80601e0 2023-10-12T22:35:39.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.793+0000 7fcabca21700 1 -- 172.21.15.195:0/3217671072 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab81092a0 con 0x7fcab80601e0 2023-10-12T22:35:39.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.794+0000 7fcaaffff700 1 -- 172.21.15.195:0/3217671072 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcaa0014420 con 0x7fcab80601e0 2023-10-12T22:35:39.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.794+0000 7fcabca21700 1 -- 172.21.15.195:0/3217671072 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab8045ad0 con 0x7fcab80601e0 2023-10-12T22:35:39.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.794+0000 7fcaaffff700 1 --2- 172.21.15.195:0/3217671072 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcaa402ff60 0x7fcaa4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:39.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.794+0000 7fcaaffff700 1 -- 172.21.15.195:0/3217671072 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcaa0041000 con 0x7fcab80601e0 2023-10-12T22:35:39.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.794+0000 7fcab6ffd700 1 --2- 172.21.15.195:0/3217671072 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcaa402ff60 0x7fcaa4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:39.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.794+0000 7fcab6ffd700 1 --2- 172.21.15.195:0/3217671072 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcaa402ff60 0x7fcaa4032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fcaa80096a0 tx=0x7fcaa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:39.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:39.798+0000 7fcaaffff700 1 -- 172.21.15.195:0/3217671072 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcaa00095f0 con 0x7fcab80601e0 2023-10-12T22:35:40.003 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:40.002+0000 7fcabca21700 1 -- 172.21.15.195:0/3217671072 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcab810a150 con 0x7fcab80601e0 2023-10-12T22:35:40.004 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:40.003+0000 7fcaaffff700 1 -- 172.21.15.195:0/3217671072 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcaa001db70 con 0x7fcab80601e0 2023-10-12T22:35:40.004 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:40.005 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:40.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:40.005+0000 7fcabca21700 1 -- 172.21.15.195:0/3217671072 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcaa402ff60 msgr2=0x7fcaa4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:40.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:40.005+0000 7fcabca21700 1 --2- 172.21.15.195:0/3217671072 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcaa402ff60 0x7fcaa4032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fcaa80096a0 tx=0x7fcaa8006b40).stop 2023-10-12T22:35:40.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:40.005+0000 7fcabca21700 1 -- 172.21.15.195:0/3217671072 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcab80601e0 msgr2=0x7fcab8108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:40.006 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:40.005+0000 7fcabca21700 1 --2- 172.21.15.195:0/3217671072 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcab80601e0 0x7fcab8108720 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fcaa0013510 tx=0x7fcaa000b040).stop 2023-10-12T22:35:40.007 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:40.005+0000 7fcabca21700 1 -- 172.21.15.195:0/3217671072 shutdown_connections 2023-10-12T22:35:40.007 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:40.005+0000 7fcabca21700 1 --2- 172.21.15.195:0/3217671072 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcaa402ff60 0x7fcaa4032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:40.007 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:40.005+0000 7fcabca21700 1 --2- 172.21.15.195:0/3217671072 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcab80601e0 0x7fcab8108720 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:40.007 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:40.005+0000 7fcabca21700 1 -- 172.21.15.195:0/3217671072 >> 172.21.15.195:0/3217671072 conn(0x7fcab80f09a0 msgr2=0x7fcab80f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:40.007 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:40.005+0000 7fcabca21700 1 -- 172.21.15.195:0/3217671072 shutdown_connections 2023-10-12T22:35:40.008 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:40.005+0000 7fcabca21700 1 -- 172.21.15.195:0/3217671072 wait complete. 2023-10-12T22:35:40.008 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:40.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:39 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:38.837006+0000 mgr.smithi131.efktma (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:41.400 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:41.401 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:41.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:40 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:40.004344+0000 mon.smithi131 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.195:0/3217671072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:42.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:40.837357+0000 mgr.smithi131.efktma (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:42.552 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.549+0000 7eff75aab700 1 -- 172.21.15.195:0/435859813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff700f6990 msgr2=0x7eff700f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:42.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.549+0000 7eff75aab700 1 --2- 172.21.15.195:0/435859813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff700f6990 0x7eff700f6db0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7eff60009fb0 tx=0x7eff60009b30).stop 2023-10-12T22:35:42.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.549+0000 7eff75aab700 1 -- 172.21.15.195:0/435859813 shutdown_connections 2023-10-12T22:35:42.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.549+0000 7eff75aab700 1 --2- 172.21.15.195:0/435859813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff700f6990 0x7eff700f6db0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:42.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.549+0000 7eff75aab700 1 -- 172.21.15.195:0/435859813 >> 172.21.15.195:0/435859813 conn(0x7eff700f0260 msgr2=0x7eff700f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:42.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.549+0000 7eff75aab700 1 -- 172.21.15.195:0/435859813 shutdown_connections 2023-10-12T22:35:42.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.549+0000 7eff75aab700 1 -- 172.21.15.195:0/435859813 wait complete. 2023-10-12T22:35:42.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.550+0000 7eff75aab700 1 Processor -- start 2023-10-12T22:35:42.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.550+0000 7eff75aab700 1 -- start start 2023-10-12T22:35:42.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.550+0000 7eff75aab700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff700f6990 0x7eff70106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:42.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.550+0000 7eff75aab700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff70106c10 con 0x7eff700f6990 2023-10-12T22:35:42.555 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.551+0000 7eff6f7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff700f6990 0x7eff70106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:42.555 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.551+0000 7eff6f7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff700f6990 0x7eff70106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:43270/0 (socket says 172.21.15.195:43270) 2023-10-12T22:35:42.555 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.551+0000 7eff6f7fe700 1 -- 172.21.15.195:0/4162311746 learned_addr learned my addr 172.21.15.195:0/4162311746 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:42.555 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.551+0000 7eff6f7fe700 1 -- 172.21.15.195:0/4162311746 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff60005040 con 0x7eff700f6990 2023-10-12T22:35:42.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.551+0000 7eff6f7fe700 1 --2- 172.21.15.195:0/4162311746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff700f6990 0x7eff70106720 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7eff60013510 tx=0x7eff6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:42.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.551+0000 7eff57fff700 1 -- 172.21.15.195:0/4162311746 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff60009320 con 0x7eff700f6990 2023-10-12T22:35:42.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.551+0000 7eff57fff700 1 -- 172.21.15.195:0/4162311746 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff60013f00 con 0x7eff700f6990 2023-10-12T22:35:42.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.551+0000 7eff75aab700 1 -- 172.21.15.195:0/4162311746 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff70106e10 con 0x7eff700f6990 2023-10-12T22:35:42.557 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.551+0000 7eff57fff700 1 -- 172.21.15.195:0/4162311746 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff60014d90 con 0x7eff700f6990 2023-10-12T22:35:42.557 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.551+0000 7eff75aab700 1 -- 172.21.15.195:0/4162311746 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff701072a0 con 0x7eff700f6990 2023-10-12T22:35:42.558 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.552+0000 7eff57fff700 1 -- 172.21.15.195:0/4162311746 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff600263a0 con 0x7eff700f6990 2023-10-12T22:35:42.558 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.552+0000 7eff57fff700 1 --2- 172.21.15.195:0/4162311746 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7eff5802ffb0 0x7eff58032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:42.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.552+0000 7eff57fff700 1 -- 172.21.15.195:0/4162311746 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff60040fd0 con 0x7eff700f6990 2023-10-12T22:35:42.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.552+0000 7eff75aab700 1 -- 172.21.15.195:0/4162311746 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff70059100 con 0x7eff700f6990 2023-10-12T22:35:42.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.552+0000 7eff6effd700 1 --2- 172.21.15.195:0/4162311746 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7eff5802ffb0 0x7eff58032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:42.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.553+0000 7eff6effd700 1 --2- 172.21.15.195:0/4162311746 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7eff5802ffb0 0x7eff58032470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7eff640096a0 tx=0x7eff64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:42.559 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.557+0000 7eff57fff700 1 -- 172.21.15.195:0/4162311746 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff60012920 con 0x7eff700f6990 2023-10-12T22:35:42.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.763+0000 7eff75aab700 1 -- 172.21.15.195:0/4162311746 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff70045ad0 con 0x7eff700f6990 2023-10-12T22:35:42.765 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.764+0000 7eff57fff700 1 -- 172.21.15.195:0/4162311746 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7eff6002b020 con 0x7eff700f6990 2023-10-12T22:35:42.765 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:42.766 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:42.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.765+0000 7eff75aab700 1 -- 172.21.15.195:0/4162311746 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7eff5802ffb0 msgr2=0x7eff58032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:42.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.765+0000 7eff75aab700 1 --2- 172.21.15.195:0/4162311746 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7eff5802ffb0 0x7eff58032470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7eff640096a0 tx=0x7eff64006b40).stop 2023-10-12T22:35:42.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.765+0000 7eff75aab700 1 -- 172.21.15.195:0/4162311746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff700f6990 msgr2=0x7eff70106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:42.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.765+0000 7eff75aab700 1 --2- 172.21.15.195:0/4162311746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff700f6990 0x7eff70106720 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7eff60013510 tx=0x7eff6000b040).stop 2023-10-12T22:35:42.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.765+0000 7eff75aab700 1 -- 172.21.15.195:0/4162311746 shutdown_connections 2023-10-12T22:35:42.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.765+0000 7eff75aab700 1 --2- 172.21.15.195:0/4162311746 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7eff5802ffb0 0x7eff58032470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:42.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.765+0000 7eff75aab700 1 --2- 172.21.15.195:0/4162311746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff700f6990 0x7eff70106720 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:42.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.765+0000 7eff75aab700 1 -- 172.21.15.195:0/4162311746 >> 172.21.15.195:0/4162311746 conn(0x7eff700f0260 msgr2=0x7eff700f4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:42.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.766+0000 7eff75aab700 1 -- 172.21.15.195:0/4162311746 shutdown_connections 2023-10-12T22:35:42.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:42.766+0000 7eff75aab700 1 -- 172.21.15.195:0/4162311746 wait complete. 2023-10-12T22:35:42.769 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:43.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:42 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:42.765161+0000 mon.smithi131 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.195:0/4162311746' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:44.149 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:44.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:44.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:43 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:42.837608+0000 mgr.smithi131.efktma (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:45.338 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.336+0000 7f4df7618700 1 -- 172.21.15.195:0/2773314816 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4df00f5410 msgr2=0x7f4df00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:45.338 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.336+0000 7f4df7618700 1 --2- 172.21.15.195:0/2773314816 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4df00f5410 0x7f4df00f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f4de4009fb0 tx=0x7f4de4009b30).stop 2023-10-12T22:35:45.338 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.336+0000 7f4df7618700 1 -- 172.21.15.195:0/2773314816 shutdown_connections 2023-10-12T22:35:45.339 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.336+0000 7f4df7618700 1 --2- 172.21.15.195:0/2773314816 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4df00f5410 0x7f4df00f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:45.339 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.336+0000 7f4df7618700 1 -- 172.21.15.195:0/2773314816 >> 172.21.15.195:0/2773314816 conn(0x7f4df00f0b70 msgr2=0x7f4df00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:45.339 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.336+0000 7f4df7618700 1 -- 172.21.15.195:0/2773314816 shutdown_connections 2023-10-12T22:35:45.339 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.336+0000 7f4df7618700 1 -- 172.21.15.195:0/2773314816 wait complete. 2023-10-12T22:35:45.339 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.337+0000 7f4df7618700 1 Processor -- start 2023-10-12T22:35:45.340 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.337+0000 7f4df7618700 1 -- start start 2023-10-12T22:35:45.340 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.337+0000 7f4df7618700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4df00f5410 0x7f4df0105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:45.340 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.337+0000 7f4df7618700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df0105af0 con 0x7f4df00f5410 2023-10-12T22:35:45.343 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.337+0000 7f4df53b4700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4df00f5410 0x7f4df0105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:45.343 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.337+0000 7f4df53b4700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4df00f5410 0x7f4df0105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:43290/0 (socket says 172.21.15.195:43290) 2023-10-12T22:35:45.343 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.337+0000 7f4df53b4700 1 -- 172.21.15.195:0/149288774 learned_addr learned my addr 172.21.15.195:0/149288774 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:45.343 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.338+0000 7f4df53b4700 1 -- 172.21.15.195:0/149288774 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de4005040 con 0x7f4df00f5410 2023-10-12T22:35:45.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.338+0000 7f4df53b4700 1 --2- 172.21.15.195:0/149288774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4df00f5410 0x7f4df0105600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f4de4013510 tx=0x7f4de400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:45.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.338+0000 7f4de1ffb700 1 -- 172.21.15.195:0/149288774 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de4009320 con 0x7f4df00f5410 2023-10-12T22:35:45.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.338+0000 7f4de1ffb700 1 -- 172.21.15.195:0/149288774 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4de4013f00 con 0x7f4df00f5410 2023-10-12T22:35:45.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.338+0000 7f4df7618700 1 -- 172.21.15.195:0/149288774 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df0105cf0 con 0x7f4df00f5410 2023-10-12T22:35:45.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.338+0000 7f4de1ffb700 1 -- 172.21.15.195:0/149288774 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de4014d90 con 0x7f4df00f5410 2023-10-12T22:35:45.345 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.338+0000 7f4df7618700 1 -- 172.21.15.195:0/149288774 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df0106180 con 0x7f4df00f5410 2023-10-12T22:35:45.345 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.339+0000 7f4de1ffb700 1 -- 172.21.15.195:0/149288774 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4de4014420 con 0x7f4df00f5410 2023-10-12T22:35:45.345 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.339+0000 7f4df7618700 1 -- 172.21.15.195:0/149288774 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df0045ad0 con 0x7f4df00f5410 2023-10-12T22:35:45.345 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.339+0000 7f4de1ffb700 1 --2- 172.21.15.195:0/149288774 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4ddc02ff60 0x7f4ddc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:45.345 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.339+0000 7f4de1ffb700 1 -- 172.21.15.195:0/149288774 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4de40410c0 con 0x7f4df00f5410 2023-10-12T22:35:45.346 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.339+0000 7f4df4bb3700 1 --2- 172.21.15.195:0/149288774 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4ddc02ff60 0x7f4ddc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:45.346 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.340+0000 7f4df4bb3700 1 --2- 172.21.15.195:0/149288774 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4ddc02ff60 0x7f4ddc032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f4dec0096a0 tx=0x7f4dec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:45.347 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.344+0000 7f4de1ffb700 1 -- 172.21.15.195:0/149288774 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4de40095f0 con 0x7f4df00f5410 2023-10-12T22:35:45.550 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.549+0000 7f4df7618700 1 -- 172.21.15.195:0/149288774 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4df00fae60 con 0x7f4df00f5410 2023-10-12T22:35:45.551 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.550+0000 7f4de1ffb700 1 -- 172.21.15.195:0/149288774 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4de401db70 con 0x7f4df00f5410 2023-10-12T22:35:45.551 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:45.552 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:45.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.551+0000 7f4df7618700 1 -- 172.21.15.195:0/149288774 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4ddc02ff60 msgr2=0x7f4ddc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:45.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.551+0000 7f4df7618700 1 --2- 172.21.15.195:0/149288774 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4ddc02ff60 0x7f4ddc032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f4dec0096a0 tx=0x7f4dec006b40).stop 2023-10-12T22:35:45.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.551+0000 7f4df7618700 1 -- 172.21.15.195:0/149288774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4df00f5410 msgr2=0x7f4df0105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:45.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.551+0000 7f4df7618700 1 --2- 172.21.15.195:0/149288774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4df00f5410 0x7f4df0105600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f4de4013510 tx=0x7f4de400b040).stop 2023-10-12T22:35:45.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.551+0000 7f4df7618700 1 -- 172.21.15.195:0/149288774 shutdown_connections 2023-10-12T22:35:45.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.551+0000 7f4df7618700 1 --2- 172.21.15.195:0/149288774 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4ddc02ff60 0x7f4ddc032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:45.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.551+0000 7f4df7618700 1 --2- 172.21.15.195:0/149288774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4df00f5410 0x7f4df0105600 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:45.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.551+0000 7f4df7618700 1 -- 172.21.15.195:0/149288774 >> 172.21.15.195:0/149288774 conn(0x7f4df00f0b70 msgr2=0x7f4df00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:45.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.551+0000 7f4df7618700 1 -- 172.21.15.195:0/149288774 shutdown_connections 2023-10-12T22:35:45.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:45.551+0000 7f4df7618700 1 -- 172.21.15.195:0/149288774 wait complete. 2023-10-12T22:35:45.555 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:46.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:46 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:44.837883+0000 mgr.smithi131.efktma (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:46.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:46 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:45.551003+0000 mon.smithi131 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.195:0/149288774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:46.930 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:46.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:48.091 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.089+0000 7fe378840700 1 -- 172.21.15.195:0/1546154527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3740f51a0 msgr2=0x7fe3740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:48.091 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.089+0000 7fe378840700 1 --2- 172.21.15.195:0/1546154527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3740f51a0 0x7fe3740f55c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fe364009fb0 tx=0x7fe364009b30).stop 2023-10-12T22:35:48.091 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.089+0000 7fe378840700 1 -- 172.21.15.195:0/1546154527 shutdown_connections 2023-10-12T22:35:48.091 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.089+0000 7fe378840700 1 --2- 172.21.15.195:0/1546154527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3740f51a0 0x7fe3740f55c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:48.092 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.089+0000 7fe378840700 1 -- 172.21.15.195:0/1546154527 >> 172.21.15.195:0/1546154527 conn(0x7fe3740f09a0 msgr2=0x7fe3740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:48.092 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.089+0000 7fe378840700 1 -- 172.21.15.195:0/1546154527 shutdown_connections 2023-10-12T22:35:48.092 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.089+0000 7fe378840700 1 -- 172.21.15.195:0/1546154527 wait complete. 2023-10-12T22:35:48.094 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.090+0000 7fe378840700 1 Processor -- start 2023-10-12T22:35:48.094 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.090+0000 7fe378840700 1 -- start start 2023-10-12T22:35:48.094 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.090+0000 7fe378840700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3740f51a0 0x7fe3741064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:48.095 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.090+0000 7fe378840700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe374106990 con 0x7fe3740f51a0 2023-10-12T22:35:48.095 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.090+0000 7fe372d9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3740f51a0 0x7fe3741064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:48.095 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.090+0000 7fe372d9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3740f51a0 0x7fe3741064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:43312/0 (socket says 172.21.15.195:43312) 2023-10-12T22:35:48.095 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.090+0000 7fe372d9d700 1 -- 172.21.15.195:0/2106876849 learned_addr learned my addr 172.21.15.195:0/2106876849 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:48.095 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.091+0000 7fe372d9d700 1 -- 172.21.15.195:0/2106876849 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe364005040 con 0x7fe3740f51a0 2023-10-12T22:35:48.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.091+0000 7fe372d9d700 1 --2- 172.21.15.195:0/2106876849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3740f51a0 0x7fe3741064a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fe364013510 tx=0x7fe36400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:48.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.091+0000 7fe36b7fe700 1 -- 172.21.15.195:0/2106876849 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe364009320 con 0x7fe3740f51a0 2023-10-12T22:35:48.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.091+0000 7fe36b7fe700 1 -- 172.21.15.195:0/2106876849 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe364013f00 con 0x7fe3740f51a0 2023-10-12T22:35:48.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.091+0000 7fe378840700 1 -- 172.21.15.195:0/2106876849 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe374106b90 con 0x7fe3740f51a0 2023-10-12T22:35:48.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.091+0000 7fe36b7fe700 1 -- 172.21.15.195:0/2106876849 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe364014dc0 con 0x7fe3740f51a0 2023-10-12T22:35:48.097 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.091+0000 7fe378840700 1 -- 172.21.15.195:0/2106876849 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe374107020 con 0x7fe3740f51a0 2023-10-12T22:35:48.097 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.092+0000 7fe36b7fe700 1 -- 172.21.15.195:0/2106876849 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3640095f0 con 0x7fe3740f51a0 2023-10-12T22:35:48.097 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.092+0000 7fe378840700 1 -- 172.21.15.195:0/2106876849 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe374045ad0 con 0x7fe3740f51a0 2023-10-12T22:35:48.097 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.092+0000 7fe36b7fe700 1 --2- 172.21.15.195:0/2106876849 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe36002ff60 0x7fe360032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:48.097 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.092+0000 7fe36b7fe700 1 -- 172.21.15.195:0/2106876849 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe364041030 con 0x7fe3740f51a0 2023-10-12T22:35:48.098 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.092+0000 7fe37259c700 1 --2- 172.21.15.195:0/2106876849 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe36002ff60 0x7fe360032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:48.098 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.092+0000 7fe37259c700 1 --2- 172.21.15.195:0/2106876849 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe36002ff60 0x7fe360032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe35c0096a0 tx=0x7fe35c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:48.098 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.097+0000 7fe36b7fe700 1 -- 172.21.15.195:0/2106876849 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe364014420 con 0x7fe3740f51a0 2023-10-12T22:35:48.307 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.305+0000 7fe378840700 1 -- 172.21.15.195:0/2106876849 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe374107ed0 con 0x7fe3740f51a0 2023-10-12T22:35:48.308 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.306+0000 7fe36b7fe700 1 -- 172.21.15.195:0/2106876849 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe364010230 con 0x7fe3740f51a0 2023-10-12T22:35:48.308 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:48.308 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:48.309 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.307+0000 7fe378840700 1 -- 172.21.15.195:0/2106876849 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe36002ff60 msgr2=0x7fe360032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:48.310 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.307+0000 7fe378840700 1 --2- 172.21.15.195:0/2106876849 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe36002ff60 0x7fe360032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe35c0096a0 tx=0x7fe35c006b40).stop 2023-10-12T22:35:48.310 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.307+0000 7fe378840700 1 -- 172.21.15.195:0/2106876849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3740f51a0 msgr2=0x7fe3741064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:48.310 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.307+0000 7fe378840700 1 --2- 172.21.15.195:0/2106876849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3740f51a0 0x7fe3741064a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fe364013510 tx=0x7fe36400b040).stop 2023-10-12T22:35:48.310 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.307+0000 7fe378840700 1 -- 172.21.15.195:0/2106876849 shutdown_connections 2023-10-12T22:35:48.310 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.307+0000 7fe378840700 1 --2- 172.21.15.195:0/2106876849 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe36002ff60 0x7fe360032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:48.311 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.307+0000 7fe378840700 1 --2- 172.21.15.195:0/2106876849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe3740f51a0 0x7fe3741064a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:48.311 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.307+0000 7fe378840700 1 -- 172.21.15.195:0/2106876849 >> 172.21.15.195:0/2106876849 conn(0x7fe3740f09a0 msgr2=0x7fe37405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:48.311 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.308+0000 7fe378840700 1 -- 172.21.15.195:0/2106876849 shutdown_connections 2023-10-12T22:35:48.311 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:48.308+0000 7fe378840700 1 -- 172.21.15.195:0/2106876849 wait complete. 2023-10-12T22:35:48.311 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:48.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:48 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:46.838229+0000 mgr.smithi131.efktma (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:49.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:49 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:48.307418+0000 mon.smithi131 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.195:0/2106876849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:49.699 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:49.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:50.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:48.838554+0000 mgr.smithi131.efktma (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:50.887 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.886+0000 7fe07baea700 1 -- 172.21.15.195:0/2573654564 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0740f5410 msgr2=0x7fe0740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:50.888 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.886+0000 7fe07baea700 1 --2- 172.21.15.195:0/2573654564 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0740f5410 0x7fe0740f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fe064009fb0 tx=0x7fe064009b30).stop 2023-10-12T22:35:50.888 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.886+0000 7fe07baea700 1 -- 172.21.15.195:0/2573654564 shutdown_connections 2023-10-12T22:35:50.888 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.886+0000 7fe07baea700 1 --2- 172.21.15.195:0/2573654564 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0740f5410 0x7fe0740f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:50.888 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.886+0000 7fe07baea700 1 -- 172.21.15.195:0/2573654564 >> 172.21.15.195:0/2573654564 conn(0x7fe0740f0b70 msgr2=0x7fe0740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:50.888 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.886+0000 7fe07baea700 1 -- 172.21.15.195:0/2573654564 shutdown_connections 2023-10-12T22:35:50.889 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.886+0000 7fe07baea700 1 -- 172.21.15.195:0/2573654564 wait complete. 2023-10-12T22:35:50.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.886+0000 7fe07baea700 1 Processor -- start 2023-10-12T22:35:50.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.887+0000 7fe07baea700 1 -- start start 2023-10-12T22:35:50.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.887+0000 7fe07baea700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0740f5410 0x7fe074109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:50.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.887+0000 7fe07baea700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe074060240 con 0x7fe0740f5410 2023-10-12T22:35:50.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.887+0000 7fe079886700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0740f5410 0x7fe074109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:50.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.887+0000 7fe079886700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0740f5410 0x7fe074109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:33052/0 (socket says 172.21.15.195:33052) 2023-10-12T22:35:50.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.887+0000 7fe079886700 1 -- 172.21.15.195:0/1060608986 learned_addr learned my addr 172.21.15.195:0/1060608986 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:50.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.887+0000 7fe079886700 1 -- 172.21.15.195:0/1060608986 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe064005040 con 0x7fe0740f5410 2023-10-12T22:35:50.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.887+0000 7fe079886700 1 --2- 172.21.15.195:0/1060608986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0740f5410 0x7fe074109430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fe064014000 tx=0x7fe06400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:50.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.888+0000 7fe06a7fc700 1 -- 172.21.15.195:0/1060608986 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0640095f0 con 0x7fe0740f5410 2023-10-12T22:35:50.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.888+0000 7fe07baea700 1 -- 172.21.15.195:0/1060608986 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe074060440 con 0x7fe0740f5410 2023-10-12T22:35:50.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.888+0000 7fe06a7fc700 1 -- 172.21.15.195:0/1060608986 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe064007e70 con 0x7fe0740f5410 2023-10-12T22:35:50.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.888+0000 7fe06a7fc700 1 -- 172.21.15.195:0/1060608986 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe064014ea0 con 0x7fe0740f5410 2023-10-12T22:35:50.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.888+0000 7fe07baea700 1 -- 172.21.15.195:0/1060608986 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe074109ae0 con 0x7fe0740f5410 2023-10-12T22:35:50.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.888+0000 7fe06a7fc700 1 -- 172.21.15.195:0/1060608986 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe064014690 con 0x7fe0740f5410 2023-10-12T22:35:50.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.888+0000 7fe07baea700 1 -- 172.21.15.195:0/1060608986 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe074045ad0 con 0x7fe0740f5410 2023-10-12T22:35:50.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.889+0000 7fe06a7fc700 1 --2- 172.21.15.195:0/1060608986 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe06002ff60 0x7fe060032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:50.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.889+0000 7fe06a7fc700 1 -- 172.21.15.195:0/1060608986 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe064040870 con 0x7fe0740f5410 2023-10-12T22:35:50.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.889+0000 7fe079085700 1 --2- 172.21.15.195:0/1060608986 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe06002ff60 0x7fe060032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:50.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.889+0000 7fe079085700 1 --2- 172.21.15.195:0/1060608986 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe06002ff60 0x7fe060032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fe0700096a0 tx=0x7fe070006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:50.895 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:50.893+0000 7fe06a7fc700 1 -- 172.21.15.195:0/1060608986 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe06400f300 con 0x7fe0740f5410 2023-10-12T22:35:51.098 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:51.097+0000 7fe07baea700 1 -- 172.21.15.195:0/1060608986 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0740fae60 con 0x7fe0740f5410 2023-10-12T22:35:51.099 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:51.098+0000 7fe06a7fc700 1 -- 172.21.15.195:0/1060608986 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe06400f300 con 0x7fe0740f5410 2023-10-12T22:35:51.099 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:51.100 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:51.101 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:51.099+0000 7fe07baea700 1 -- 172.21.15.195:0/1060608986 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe06002ff60 msgr2=0x7fe060032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:51.101 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:51.099+0000 7fe07baea700 1 --2- 172.21.15.195:0/1060608986 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe06002ff60 0x7fe060032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fe0700096a0 tx=0x7fe070006b40).stop 2023-10-12T22:35:51.102 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:51.099+0000 7fe07baea700 1 -- 172.21.15.195:0/1060608986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0740f5410 msgr2=0x7fe074109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:51.102 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:51.099+0000 7fe07baea700 1 --2- 172.21.15.195:0/1060608986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0740f5410 0x7fe074109430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fe064014000 tx=0x7fe06400b040).stop 2023-10-12T22:35:51.102 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:51.099+0000 7fe07baea700 1 -- 172.21.15.195:0/1060608986 shutdown_connections 2023-10-12T22:35:51.102 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:51.100+0000 7fe07baea700 1 --2- 172.21.15.195:0/1060608986 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe06002ff60 0x7fe060032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:51.102 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:51.100+0000 7fe07baea700 1 --2- 172.21.15.195:0/1060608986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0740f5410 0x7fe074109430 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:51.103 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:51.100+0000 7fe07baea700 1 -- 172.21.15.195:0/1060608986 >> 172.21.15.195:0/1060608986 conn(0x7fe0740f0b70 msgr2=0x7fe0740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:51.103 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:51.100+0000 7fe07baea700 1 -- 172.21.15.195:0/1060608986 shutdown_connections 2023-10-12T22:35:51.103 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:51.100+0000 7fe07baea700 1 -- 172.21.15.195:0/1060608986 wait complete. 2023-10-12T22:35:51.103 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:52.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:50.838916+0000 mgr.smithi131.efktma (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:52.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:51.099302+0000 mon.smithi131 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.195:0/1060608986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:52.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:52.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:53.642 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.640+0000 7f63b047c700 1 -- 172.21.15.195:0/545615934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63a80f6510 msgr2=0x7f63a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:53.642 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.640+0000 7f63b047c700 1 --2- 172.21.15.195:0/545615934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63a80f6510 0x7f63a80f6930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f639c009fb0 tx=0x7f639c009b30).stop 2023-10-12T22:35:53.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.640+0000 7f63b047c700 1 -- 172.21.15.195:0/545615934 shutdown_connections 2023-10-12T22:35:53.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.640+0000 7f63b047c700 1 --2- 172.21.15.195:0/545615934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63a80f6510 0x7f63a80f6930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:53.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.640+0000 7f63b047c700 1 -- 172.21.15.195:0/545615934 >> 172.21.15.195:0/545615934 conn(0x7f63a80f1c70 msgr2=0x7f63a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:53.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.641+0000 7f63b047c700 1 -- 172.21.15.195:0/545615934 shutdown_connections 2023-10-12T22:35:53.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.641+0000 7f63b047c700 1 -- 172.21.15.195:0/545615934 wait complete. 2023-10-12T22:35:53.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.641+0000 7f63b047c700 1 Processor -- start 2023-10-12T22:35:53.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.641+0000 7f63b047c700 1 -- start start 2023-10-12T22:35:53.646 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.641+0000 7f63b047c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63a80f6510 0x7f63a8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:53.646 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.641+0000 7f63b047c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a8106c20 con 0x7f63a80f6510 2023-10-12T22:35:53.646 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.642+0000 7f63ae218700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63a80f6510 0x7f63a8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:53.646 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.642+0000 7f63ae218700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63a80f6510 0x7f63a8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:33060/0 (socket says 172.21.15.195:33060) 2023-10-12T22:35:53.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.642+0000 7f63ae218700 1 -- 172.21.15.195:0/3874846663 learned_addr learned my addr 172.21.15.195:0/3874846663 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:53.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.642+0000 7f63ae218700 1 -- 172.21.15.195:0/3874846663 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c005040 con 0x7f63a80f6510 2023-10-12T22:35:53.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.642+0000 7f63ae218700 1 --2- 172.21.15.195:0/3874846663 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63a80f6510 0x7f63a8106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f639c013510 tx=0x7f639c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:53.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.642+0000 7f639affd700 1 -- 172.21.15.195:0/3874846663 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f639c009320 con 0x7f63a80f6510 2023-10-12T22:35:53.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.642+0000 7f63b047c700 1 -- 172.21.15.195:0/3874846663 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a8106e20 con 0x7f63a80f6510 2023-10-12T22:35:53.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.642+0000 7f639affd700 1 -- 172.21.15.195:0/3874846663 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f639c013f00 con 0x7f63a80f6510 2023-10-12T22:35:53.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.642+0000 7f639affd700 1 -- 172.21.15.195:0/3874846663 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f639c014d90 con 0x7f63a80f6510 2023-10-12T22:35:53.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.642+0000 7f63b047c700 1 -- 172.21.15.195:0/3874846663 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a81072b0 con 0x7f63a80f6510 2023-10-12T22:35:53.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.643+0000 7f639affd700 1 -- 172.21.15.195:0/3874846663 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f639c0263a0 con 0x7f63a80f6510 2023-10-12T22:35:53.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.643+0000 7f639affd700 1 --2- 172.21.15.195:0/3874846663 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f639402ff60 0x7f6394032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:53.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.643+0000 7f63b047c700 1 -- 172.21.15.195:0/3874846663 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a8045ad0 con 0x7f63a80f6510 2023-10-12T22:35:53.649 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.643+0000 7f639affd700 1 -- 172.21.15.195:0/3874846663 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f639c041000 con 0x7f63a80f6510 2023-10-12T22:35:53.649 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.643+0000 7f63ada17700 1 --2- 172.21.15.195:0/3874846663 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f639402ff60 0x7f6394032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:53.649 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.644+0000 7f63ada17700 1 --2- 172.21.15.195:0/3874846663 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f639402ff60 0x7f6394032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f63a40096a0 tx=0x7f63a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:53.650 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.648+0000 7f639affd700 1 -- 172.21.15.195:0/3874846663 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f639c012920 con 0x7f63a80f6510 2023-10-12T22:35:53.860 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.859+0000 7f63b047c700 1 -- 172.21.15.195:0/3874846663 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63a80fc050 con 0x7f63a80f6510 2023-10-12T22:35:53.861 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.859+0000 7f639affd700 1 -- 172.21.15.195:0/3874846663 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f639c02b020 con 0x7f63a80f6510 2023-10-12T22:35:53.861 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:53.861 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:53.862 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.861+0000 7f63b047c700 1 -- 172.21.15.195:0/3874846663 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f639402ff60 msgr2=0x7f6394032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:53.862 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.861+0000 7f63b047c700 1 --2- 172.21.15.195:0/3874846663 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f639402ff60 0x7f6394032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f63a40096a0 tx=0x7f63a4006b40).stop 2023-10-12T22:35:53.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.861+0000 7f63b047c700 1 -- 172.21.15.195:0/3874846663 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63a80f6510 msgr2=0x7f63a8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:53.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.861+0000 7f63b047c700 1 --2- 172.21.15.195:0/3874846663 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63a80f6510 0x7f63a8106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f639c013510 tx=0x7f639c00b040).stop 2023-10-12T22:35:53.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.861+0000 7f63b047c700 1 -- 172.21.15.195:0/3874846663 shutdown_connections 2023-10-12T22:35:53.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.861+0000 7f63b047c700 1 --2- 172.21.15.195:0/3874846663 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f639402ff60 0x7f6394032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:53.863 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.861+0000 7f63b047c700 1 --2- 172.21.15.195:0/3874846663 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63a80f6510 0x7f63a8106730 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:53.864 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.861+0000 7f63b047c700 1 -- 172.21.15.195:0/3874846663 >> 172.21.15.195:0/3874846663 conn(0x7f63a80f1c70 msgr2=0x7f63a80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:53.864 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.861+0000 7f63b047c700 1 -- 172.21.15.195:0/3874846663 shutdown_connections 2023-10-12T22:35:53.864 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:53.861+0000 7f63b047c700 1 -- 172.21.15.195:0/3874846663 wait complete. 2023-10-12T22:35:53.865 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:54.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:52.839245+0000 mgr.smithi131.efktma (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:54.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:53.860744+0000 mon.smithi131 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.195:0/3874846663' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:55.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:55.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:56.395 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.394+0000 7f14e4d90700 1 -- 172.21.15.195:0/437707836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14e00f6510 msgr2=0x7f14e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:56.396 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.394+0000 7f14e4d90700 1 --2- 172.21.15.195:0/437707836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14e00f6510 0x7f14e00f6930 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f14d0009fb0 tx=0x7f14d0009b30).stop 2023-10-12T22:35:56.396 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.394+0000 7f14e4d90700 1 -- 172.21.15.195:0/437707836 shutdown_connections 2023-10-12T22:35:56.396 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.394+0000 7f14e4d90700 1 --2- 172.21.15.195:0/437707836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14e00f6510 0x7f14e00f6930 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:56.396 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.394+0000 7f14e4d90700 1 -- 172.21.15.195:0/437707836 >> 172.21.15.195:0/437707836 conn(0x7f14e00f1c70 msgr2=0x7f14e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:56.397 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.394+0000 7f14e4d90700 1 -- 172.21.15.195:0/437707836 shutdown_connections 2023-10-12T22:35:56.397 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.394+0000 7f14e4d90700 1 -- 172.21.15.195:0/437707836 wait complete. 2023-10-12T22:35:56.397 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.394+0000 7f14e4d90700 1 Processor -- start 2023-10-12T22:35:56.397 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.394+0000 7f14e4d90700 1 -- start start 2023-10-12T22:35:56.397 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.394+0000 7f14e4d90700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14e00f6510 0x7f14e0108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:56.398 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.394+0000 7f14e4d90700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e0108e00 con 0x7f14e00f6510 2023-10-12T22:35:56.399 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.395+0000 7f14de59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14e00f6510 0x7f14e0108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:56.399 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.395+0000 7f14de59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14e00f6510 0x7f14e0108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:33068/0 (socket says 172.21.15.195:33068) 2023-10-12T22:35:56.400 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.395+0000 7f14de59c700 1 -- 172.21.15.195:0/4130620703 learned_addr learned my addr 172.21.15.195:0/4130620703 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:56.400 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.395+0000 7f14de59c700 1 -- 172.21.15.195:0/4130620703 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d0005040 con 0x7f14e00f6510 2023-10-12T22:35:56.400 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.395+0000 7f14de59c700 1 --2- 172.21.15.195:0/4130620703 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14e00f6510 0x7f14e0108910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f14d0013510 tx=0x7f14d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:56.400 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.395+0000 7f14d6ffd700 1 -- 172.21.15.195:0/4130620703 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14d0009320 con 0x7f14e00f6510 2023-10-12T22:35:56.400 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.395+0000 7f14e4d90700 1 -- 172.21.15.195:0/4130620703 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e0109000 con 0x7f14e00f6510 2023-10-12T22:35:56.401 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.395+0000 7f14d6ffd700 1 -- 172.21.15.195:0/4130620703 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14d0013f00 con 0x7f14e00f6510 2023-10-12T22:35:56.401 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.395+0000 7f14d6ffd700 1 -- 172.21.15.195:0/4130620703 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14d0014d90 con 0x7f14e00f6510 2023-10-12T22:35:56.401 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.396+0000 7f14e4d90700 1 -- 172.21.15.195:0/4130620703 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e0109490 con 0x7f14e00f6510 2023-10-12T22:35:56.401 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.396+0000 7f14d6ffd700 1 -- 172.21.15.195:0/4130620703 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14d0014420 con 0x7f14e00f6510 2023-10-12T22:35:56.401 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.396+0000 7f14e4d90700 1 -- 172.21.15.195:0/4130620703 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e0045ad0 con 0x7f14e00f6510 2023-10-12T22:35:56.402 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.397+0000 7f14d6ffd700 1 --2- 172.21.15.195:0/4130620703 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f14cc02ff60 0x7f14cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:56.402 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.397+0000 7f14d6ffd700 1 -- 172.21.15.195:0/4130620703 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14d0040ff0 con 0x7f14e00f6510 2023-10-12T22:35:56.402 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.397+0000 7f14ddd9b700 1 --2- 172.21.15.195:0/4130620703 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f14cc02ff60 0x7f14cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:56.402 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.397+0000 7f14ddd9b700 1 --2- 172.21.15.195:0/4130620703 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f14cc02ff60 0x7f14cc032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f14c80096a0 tx=0x7f14c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:56.403 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.401+0000 7f14d6ffd700 1 -- 172.21.15.195:0/4130620703 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14d00095f0 con 0x7f14e00f6510 2023-10-12T22:35:56.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:54.839549+0000 mgr.smithi131.efktma (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:56.610 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.609+0000 7f14e4d90700 1 -- 172.21.15.195:0/4130620703 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14e010a340 con 0x7f14e00f6510 2023-10-12T22:35:56.611 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.610+0000 7f14d6ffd700 1 -- 172.21.15.195:0/4130620703 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f14d001db70 con 0x7f14e00f6510 2023-10-12T22:35:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:56.611 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:56.612 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.611+0000 7f14e4d90700 1 -- 172.21.15.195:0/4130620703 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f14cc02ff60 msgr2=0x7f14cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:56.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.611+0000 7f14e4d90700 1 --2- 172.21.15.195:0/4130620703 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f14cc02ff60 0x7f14cc032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f14c80096a0 tx=0x7f14c8006b40).stop 2023-10-12T22:35:56.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.611+0000 7f14e4d90700 1 -- 172.21.15.195:0/4130620703 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14e00f6510 msgr2=0x7f14e0108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:56.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.611+0000 7f14e4d90700 1 --2- 172.21.15.195:0/4130620703 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14e00f6510 0x7f14e0108910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f14d0013510 tx=0x7f14d000b040).stop 2023-10-12T22:35:56.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.611+0000 7f14e4d90700 1 -- 172.21.15.195:0/4130620703 shutdown_connections 2023-10-12T22:35:56.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.611+0000 7f14e4d90700 1 --2- 172.21.15.195:0/4130620703 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f14cc02ff60 0x7f14cc032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:56.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.611+0000 7f14e4d90700 1 --2- 172.21.15.195:0/4130620703 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f14e00f6510 0x7f14e0108910 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:56.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.611+0000 7f14e4d90700 1 -- 172.21.15.195:0/4130620703 >> 172.21.15.195:0/4130620703 conn(0x7f14e00f1c70 msgr2=0x7f14e00f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:56.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.611+0000 7f14e4d90700 1 -- 172.21.15.195:0/4130620703 shutdown_connections 2023-10-12T22:35:56.614 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:56.611+0000 7f14e4d90700 1 -- 172.21.15.195:0/4130620703 wait complete. 2023-10-12T22:35:56.615 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:35:57.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:57 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:56.611082+0000 mon.smithi131 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.195:0/4130620703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:35:58.023 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:35:58.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:35:58.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:35:58 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:56.839854+0000 mgr.smithi131.efktma (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:35:59.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.138+0000 7f377e906700 1 -- 172.21.15.195:0/2398118793 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37780f5410 msgr2=0x7f37780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:59.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.138+0000 7f377e906700 1 --2- 172.21.15.195:0/2398118793 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37780f5410 0x7f37780f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f3768009fb0 tx=0x7f3768009b30).stop 2023-10-12T22:35:59.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.138+0000 7f377e906700 1 -- 172.21.15.195:0/2398118793 shutdown_connections 2023-10-12T22:35:59.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.138+0000 7f377e906700 1 --2- 172.21.15.195:0/2398118793 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37780f5410 0x7f37780f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:59.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.138+0000 7f377e906700 1 -- 172.21.15.195:0/2398118793 >> 172.21.15.195:0/2398118793 conn(0x7f37780f0b70 msgr2=0x7f37780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:59.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.138+0000 7f377e906700 1 -- 172.21.15.195:0/2398118793 shutdown_connections 2023-10-12T22:35:59.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.138+0000 7f377e906700 1 -- 172.21.15.195:0/2398118793 wait complete. 2023-10-12T22:35:59.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.139+0000 7f377e906700 1 Processor -- start 2023-10-12T22:35:59.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.139+0000 7f377e906700 1 -- start start 2023-10-12T22:35:59.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.139+0000 7f377e906700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37780f5410 0x7f3778108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:59.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.139+0000 7f377e906700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3778108e10 con 0x7f37780f5410 2023-10-12T22:35:59.144 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.139+0000 7f3777fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37780f5410 0x7f3778108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:59.144 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.139+0000 7f3777fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37780f5410 0x7f3778108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:42358/0 (socket says 172.21.15.195:42358) 2023-10-12T22:35:59.144 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.139+0000 7f3777fff700 1 -- 172.21.15.195:0/4203681930 learned_addr learned my addr 172.21.15.195:0/4203681930 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:35:59.144 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.140+0000 7f3777fff700 1 -- 172.21.15.195:0/4203681930 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3768005040 con 0x7f37780f5410 2023-10-12T22:35:59.145 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.140+0000 7f3777fff700 1 --2- 172.21.15.195:0/4203681930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37780f5410 0x7f3778108920 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f3768014000 tx=0x7f376800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:59.145 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.140+0000 7f3774ff9700 1 -- 172.21.15.195:0/4203681930 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37680095f0 con 0x7f37780f5410 2023-10-12T22:35:59.145 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.140+0000 7f377e906700 1 -- 172.21.15.195:0/4203681930 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3778109010 con 0x7f37780f5410 2023-10-12T22:35:59.145 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.140+0000 7f3774ff9700 1 -- 172.21.15.195:0/4203681930 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3768007e70 con 0x7f37780f5410 2023-10-12T22:35:59.146 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.140+0000 7f3774ff9700 1 -- 172.21.15.195:0/4203681930 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3768014ea0 con 0x7f37780f5410 2023-10-12T22:35:59.146 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.140+0000 7f377e906700 1 -- 172.21.15.195:0/4203681930 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37781094a0 con 0x7f37780f5410 2023-10-12T22:35:59.149 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.141+0000 7f3774ff9700 1 -- 172.21.15.195:0/4203681930 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3768025070 con 0x7f37780f5410 2023-10-12T22:35:59.149 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.141+0000 7f3774ff9700 1 --2- 172.21.15.195:0/4203681930 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f376402ff60 0x7f3764032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:35:59.149 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.141+0000 7f377e906700 1 -- 172.21.15.195:0/4203681930 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3778045ad0 con 0x7f37780f5410 2023-10-12T22:35:59.150 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.141+0000 7f3774ff9700 1 -- 172.21.15.195:0/4203681930 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3768040920 con 0x7f37780f5410 2023-10-12T22:35:59.150 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.141+0000 7f37777fe700 1 --2- 172.21.15.195:0/4203681930 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f376402ff60 0x7f3764032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:35:59.150 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.141+0000 7f37777fe700 1 --2- 172.21.15.195:0/4203681930 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f376402ff60 0x7f3764032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f37600096a0 tx=0x7f3760006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:35:59.150 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.145+0000 7f3774ff9700 1 -- 172.21.15.195:0/4203681930 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f376800f4e0 con 0x7f37780f5410 2023-10-12T22:35:59.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.354+0000 7f377e906700 1 -- 172.21.15.195:0/4203681930 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f377810a2f0 con 0x7f37780f5410 2023-10-12T22:35:59.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.355+0000 7f3774ff9700 1 -- 172.21.15.195:0/4203681930 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f376800f300 con 0x7f37780f5410 2023-10-12T22:35:59.356 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:35:59.356 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:35:59.359 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.357+0000 7f377e906700 1 -- 172.21.15.195:0/4203681930 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f376402ff60 msgr2=0x7f3764032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:59.359 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.357+0000 7f377e906700 1 --2- 172.21.15.195:0/4203681930 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f376402ff60 0x7f3764032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f37600096a0 tx=0x7f3760006b40).stop 2023-10-12T22:35:59.359 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.357+0000 7f377e906700 1 -- 172.21.15.195:0/4203681930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37780f5410 msgr2=0x7f3778108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:35:59.359 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.357+0000 7f377e906700 1 --2- 172.21.15.195:0/4203681930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37780f5410 0x7f3778108920 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f3768014000 tx=0x7f376800b040).stop 2023-10-12T22:35:59.359 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.357+0000 7f377e906700 1 -- 172.21.15.195:0/4203681930 shutdown_connections 2023-10-12T22:35:59.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.357+0000 7f377e906700 1 --2- 172.21.15.195:0/4203681930 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f376402ff60 0x7f3764032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:59.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.357+0000 7f377e906700 1 --2- 172.21.15.195:0/4203681930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f37780f5410 0x7f3778108920 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:35:59.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.357+0000 7f377e906700 1 -- 172.21.15.195:0/4203681930 >> 172.21.15.195:0/4203681930 conn(0x7f37780f0b70 msgr2=0x7f377805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:35:59.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.357+0000 7f377e906700 1 -- 172.21.15.195:0/4203681930 shutdown_connections 2023-10-12T22:35:59.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:35:59.357+0000 7f377e906700 1 -- 172.21.15.195:0/4203681930 wait complete. 2023-10-12T22:35:59.361 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:00.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:35:58.840198+0000 mgr.smithi131.efktma (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:00.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:35:59.355731+0000 mon.smithi131 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.195:0/4203681930' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:00.759 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:00.760 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:01.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.910+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/4117432046 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a080f51a0 msgr2=0x7f8a080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:01.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.910+0000 7f8a0d5aa700 1 --2- 172.21.15.195:0/4117432046 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a080f51a0 0x7f8a080f55c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f89f8009fb0 tx=0x7f89f8009b30).stop 2023-10-12T22:36:01.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.910+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/4117432046 shutdown_connections 2023-10-12T22:36:01.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.910+0000 7f8a0d5aa700 1 --2- 172.21.15.195:0/4117432046 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a080f51a0 0x7f8a080f55c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:01.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.910+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/4117432046 >> 172.21.15.195:0/4117432046 conn(0x7f8a080f09a0 msgr2=0x7f8a080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:01.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.910+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/4117432046 shutdown_connections 2023-10-12T22:36:01.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.910+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/4117432046 wait complete. 2023-10-12T22:36:01.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.910+0000 7f8a0d5aa700 1 Processor -- start 2023-10-12T22:36:01.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.911+0000 7f8a0d5aa700 1 -- start start 2023-10-12T22:36:01.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.911+0000 7f8a0d5aa700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a080f51a0 0x7f8a08109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:01.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.911+0000 7f8a0d5aa700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a08060280 con 0x7f8a080f51a0 2023-10-12T22:36:01.916 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.911+0000 7f8a07fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a080f51a0 0x7f8a08109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:01.916 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.911+0000 7f8a07fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a080f51a0 0x7f8a08109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:42378/0 (socket says 172.21.15.195:42378) 2023-10-12T22:36:01.916 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.911+0000 7f8a07fff700 1 -- 172.21.15.195:0/906938885 learned_addr learned my addr 172.21.15.195:0/906938885 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:01.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.911+0000 7f8a07fff700 1 -- 172.21.15.195:0/906938885 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f8005040 con 0x7f8a080f51a0 2023-10-12T22:36:01.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.912+0000 7f8a07fff700 1 --2- 172.21.15.195:0/906938885 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a080f51a0 0x7f8a08109170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f89f8013510 tx=0x7f89f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:01.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.912+0000 7f8a04ff9700 1 -- 172.21.15.195:0/906938885 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89f8009320 con 0x7f8a080f51a0 2023-10-12T22:36:01.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.912+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/906938885 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a08060480 con 0x7f8a080f51a0 2023-10-12T22:36:01.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.912+0000 7f8a04ff9700 1 -- 172.21.15.195:0/906938885 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89f8013f00 con 0x7f8a080f51a0 2023-10-12T22:36:01.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.912+0000 7f8a04ff9700 1 -- 172.21.15.195:0/906938885 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89f8014dc0 con 0x7f8a080f51a0 2023-10-12T22:36:01.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.912+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/906938885 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a081098a0 con 0x7f8a080f51a0 2023-10-12T22:36:01.919 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.913+0000 7f8a04ff9700 1 -- 172.21.15.195:0/906938885 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89f8014420 con 0x7f8a080f51a0 2023-10-12T22:36:01.919 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.913+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/906938885 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a08045ad0 con 0x7f8a080f51a0 2023-10-12T22:36:01.919 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.913+0000 7f8a04ff9700 1 --2- 172.21.15.195:0/906938885 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f89f002ff60 0x7f89f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:01.920 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.913+0000 7f8a04ff9700 1 -- 172.21.15.195:0/906938885 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89f8041050 con 0x7f8a080f51a0 2023-10-12T22:36:01.920 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.913+0000 7f8a077fe700 1 --2- 172.21.15.195:0/906938885 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f89f002ff60 0x7f89f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:01.920 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.914+0000 7f8a077fe700 1 --2- 172.21.15.195:0/906938885 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f89f002ff60 0x7f89f0032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f89fc0096a0 tx=0x7f89fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:01.921 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:01.918+0000 7f8a04ff9700 1 -- 172.21.15.195:0/906938885 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89f80095f0 con 0x7f8a080f51a0 2023-10-12T22:36:02.126 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:02.125+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/906938885 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a080fabe0 con 0x7f8a080f51a0 2023-10-12T22:36:02.127 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:02.126+0000 7f8a04ff9700 1 -- 172.21.15.195:0/906938885 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f89f801cb10 con 0x7f8a080f51a0 2023-10-12T22:36:02.127 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:02.127 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:02.128 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:02.127+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/906938885 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f89f002ff60 msgr2=0x7f89f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:02.129 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:02.127+0000 7f8a0d5aa700 1 --2- 172.21.15.195:0/906938885 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f89f002ff60 0x7f89f0032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f89fc0096a0 tx=0x7f89fc006b40).stop 2023-10-12T22:36:02.129 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:02.127+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/906938885 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a080f51a0 msgr2=0x7f8a08109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:02.129 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:02.127+0000 7f8a0d5aa700 1 --2- 172.21.15.195:0/906938885 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a080f51a0 0x7f8a08109170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f89f8013510 tx=0x7f89f800b040).stop 2023-10-12T22:36:02.129 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:02.127+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/906938885 shutdown_connections 2023-10-12T22:36:02.129 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:02.127+0000 7f8a0d5aa700 1 --2- 172.21.15.195:0/906938885 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f89f002ff60 0x7f89f0032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:02.130 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:02.127+0000 7f8a0d5aa700 1 --2- 172.21.15.195:0/906938885 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a080f51a0 0x7f8a08109170 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:02.130 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:02.127+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/906938885 >> 172.21.15.195:0/906938885 conn(0x7f8a080f09a0 msgr2=0x7f8a080f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:02.130 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:02.127+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/906938885 shutdown_connections 2023-10-12T22:36:02.130 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:02.127+0000 7f8a0d5aa700 1 -- 172.21.15.195:0/906938885 wait complete. 2023-10-12T22:36:02.131 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:02.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:02 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:00.840424+0000 mgr.smithi131.efktma (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:03.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:03 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:02.126938+0000 mon.smithi131 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.195:0/906938885' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:03.502 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:03.502 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:04.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:02.840747+0000 mgr.smithi131.efktma (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:04.700 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.697+0000 7fea22b45700 1 -- 172.21.15.195:0/3737490651 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea1c0f6510 msgr2=0x7fea1c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:04.701 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.697+0000 7fea22b45700 1 --2- 172.21.15.195:0/3737490651 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea1c0f6510 0x7fea1c0f6930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fea10009fb0 tx=0x7fea10009b30).stop 2023-10-12T22:36:04.701 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.697+0000 7fea22b45700 1 -- 172.21.15.195:0/3737490651 shutdown_connections 2023-10-12T22:36:04.701 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.697+0000 7fea22b45700 1 --2- 172.21.15.195:0/3737490651 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea1c0f6510 0x7fea1c0f6930 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:04.701 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.697+0000 7fea22b45700 1 -- 172.21.15.195:0/3737490651 >> 172.21.15.195:0/3737490651 conn(0x7fea1c0f1c70 msgr2=0x7fea1c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:04.701 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.697+0000 7fea22b45700 1 -- 172.21.15.195:0/3737490651 shutdown_connections 2023-10-12T22:36:04.702 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.697+0000 7fea22b45700 1 -- 172.21.15.195:0/3737490651 wait complete. 2023-10-12T22:36:04.702 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.698+0000 7fea22b45700 1 Processor -- start 2023-10-12T22:36:04.702 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.698+0000 7fea22b45700 1 -- start start 2023-10-12T22:36:04.702 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.698+0000 7fea22b45700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea1c0f6510 0x7fea1c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:04.702 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.698+0000 7fea22b45700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c069e20 con 0x7fea1c0f6510 2023-10-12T22:36:04.703 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.699+0000 7fea208e1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea1c0f6510 0x7fea1c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:04.703 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.699+0000 7fea208e1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea1c0f6510 0x7fea1c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:42400/0 (socket says 172.21.15.195:42400) 2023-10-12T22:36:04.703 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.699+0000 7fea208e1700 1 -- 172.21.15.195:0/559510180 learned_addr learned my addr 172.21.15.195:0/559510180 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:04.703 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.699+0000 7fea208e1700 1 -- 172.21.15.195:0/559510180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea10005040 con 0x7fea1c0f6510 2023-10-12T22:36:04.703 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.699+0000 7fea208e1700 1 --2- 172.21.15.195:0/559510180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea1c0f6510 0x7fea1c06b9a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fea10013510 tx=0x7fea1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:04.704 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.699+0000 7fea197fa700 1 -- 172.21.15.195:0/559510180 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea10009320 con 0x7fea1c0f6510 2023-10-12T22:36:04.704 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.699+0000 7fea22b45700 1 -- 172.21.15.195:0/559510180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1c069fc0 con 0x7fea1c0f6510 2023-10-12T22:36:04.704 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.699+0000 7fea197fa700 1 -- 172.21.15.195:0/559510180 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea10013f00 con 0x7fea1c0f6510 2023-10-12T22:36:04.704 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.699+0000 7fea197fa700 1 -- 172.21.15.195:0/559510180 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea10014d90 con 0x7fea1c0f6510 2023-10-12T22:36:04.704 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.699+0000 7fea22b45700 1 -- 172.21.15.195:0/559510180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1c06a450 con 0x7fea1c0f6510 2023-10-12T22:36:04.706 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.700+0000 7fea197fa700 1 -- 172.21.15.195:0/559510180 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea10014420 con 0x7fea1c0f6510 2023-10-12T22:36:04.706 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.700+0000 7fea22b45700 1 -- 172.21.15.195:0/559510180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1c059100 con 0x7fea1c0f6510 2023-10-12T22:36:04.706 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.700+0000 7fea197fa700 1 --2- 172.21.15.195:0/559510180 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fea0c02ff60 0x7fea0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:04.706 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.700+0000 7fea197fa700 1 -- 172.21.15.195:0/559510180 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea10041050 con 0x7fea1c0f6510 2023-10-12T22:36:04.706 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.700+0000 7fea1bfff700 1 --2- 172.21.15.195:0/559510180 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fea0c02ff60 0x7fea0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:04.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.701+0000 7fea1bfff700 1 --2- 172.21.15.195:0/559510180 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fea0c02ff60 0x7fea0c032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fea080096a0 tx=0x7fea08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:04.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.705+0000 7fea197fa700 1 -- 172.21.15.195:0/559510180 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea100095f0 con 0x7fea1c0f6510 2023-10-12T22:36:04.910 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.908+0000 7fea22b45700 1 -- 172.21.15.195:0/559510180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea1c045ad0 con 0x7fea1c0f6510 2023-10-12T22:36:04.910 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.909+0000 7fea197fa700 1 -- 172.21.15.195:0/559510180 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fea1001db70 con 0x7fea1c0f6510 2023-10-12T22:36:04.911 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:04.911 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:04.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.911+0000 7fea22b45700 1 -- 172.21.15.195:0/559510180 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fea0c02ff60 msgr2=0x7fea0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:04.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.911+0000 7fea22b45700 1 --2- 172.21.15.195:0/559510180 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fea0c02ff60 0x7fea0c032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fea080096a0 tx=0x7fea08006b40).stop 2023-10-12T22:36:04.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.911+0000 7fea22b45700 1 -- 172.21.15.195:0/559510180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea1c0f6510 msgr2=0x7fea1c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:04.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.911+0000 7fea22b45700 1 --2- 172.21.15.195:0/559510180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea1c0f6510 0x7fea1c06b9a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fea10013510 tx=0x7fea1000b040).stop 2023-10-12T22:36:04.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.911+0000 7fea22b45700 1 -- 172.21.15.195:0/559510180 shutdown_connections 2023-10-12T22:36:04.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.911+0000 7fea22b45700 1 --2- 172.21.15.195:0/559510180 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fea0c02ff60 0x7fea0c032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:04.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.911+0000 7fea22b45700 1 --2- 172.21.15.195:0/559510180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea1c0f6510 0x7fea1c06b9a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:04.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.911+0000 7fea22b45700 1 -- 172.21.15.195:0/559510180 >> 172.21.15.195:0/559510180 conn(0x7fea1c0f1c70 msgr2=0x7fea1c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:04.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.911+0000 7fea22b45700 1 -- 172.21.15.195:0/559510180 shutdown_connections 2023-10-12T22:36:04.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:04.911+0000 7fea22b45700 1 -- 172.21.15.195:0/559510180 wait complete. 2023-10-12T22:36:04.914 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:05.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:05 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:04.910548+0000 mon.smithi131 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.195:0/559510180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:06.288 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:06.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:06.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:04.841068+0000 mgr.smithi131.efktma (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:07.471 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.468+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3672821959 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8c00f5410 msgr2=0x7ff8c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:07.471 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.468+0000 7ff8c4ff8700 1 --2- 172.21.15.195:0/3672821959 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8c00f5410 0x7ff8c00f5830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7ff8b0009fb0 tx=0x7ff8b0009b30).stop 2023-10-12T22:36:07.471 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.469+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3672821959 shutdown_connections 2023-10-12T22:36:07.472 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.469+0000 7ff8c4ff8700 1 --2- 172.21.15.195:0/3672821959 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8c00f5410 0x7ff8c00f5830 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:07.472 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.469+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3672821959 >> 172.21.15.195:0/3672821959 conn(0x7ff8c00f0b70 msgr2=0x7ff8c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:07.472 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.469+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3672821959 shutdown_connections 2023-10-12T22:36:07.472 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.469+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3672821959 wait complete. 2023-10-12T22:36:07.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.469+0000 7ff8c4ff8700 1 Processor -- start 2023-10-12T22:36:07.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.469+0000 7ff8c4ff8700 1 -- start start 2023-10-12T22:36:07.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.469+0000 7ff8c4ff8700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8c00f5410 0x7ff8c0106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:07.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.469+0000 7ff8c4ff8700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c0106c10 con 0x7ff8c00f5410 2023-10-12T22:36:07.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.470+0000 7ff8be59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8c00f5410 0x7ff8c0106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:07.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.470+0000 7ff8be59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8c00f5410 0x7ff8c0106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:42408/0 (socket says 172.21.15.195:42408) 2023-10-12T22:36:07.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.470+0000 7ff8be59c700 1 -- 172.21.15.195:0/3592239420 learned_addr learned my addr 172.21.15.195:0/3592239420 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:07.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.470+0000 7ff8be59c700 1 -- 172.21.15.195:0/3592239420 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b0005040 con 0x7ff8c00f5410 2023-10-12T22:36:07.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.470+0000 7ff8be59c700 1 --2- 172.21.15.195:0/3592239420 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8c00f5410 0x7ff8c0106720 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff8b0013510 tx=0x7ff8b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:07.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.470+0000 7ff8b6ffd700 1 -- 172.21.15.195:0/3592239420 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8b0009320 con 0x7ff8c00f5410 2023-10-12T22:36:07.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.470+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3592239420 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c0106e10 con 0x7ff8c00f5410 2023-10-12T22:36:07.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.470+0000 7ff8b6ffd700 1 -- 172.21.15.195:0/3592239420 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8b0013f00 con 0x7ff8c00f5410 2023-10-12T22:36:07.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.470+0000 7ff8b6ffd700 1 -- 172.21.15.195:0/3592239420 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8b0014d90 con 0x7ff8c00f5410 2023-10-12T22:36:07.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.470+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3592239420 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c01072a0 con 0x7ff8c00f5410 2023-10-12T22:36:07.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.471+0000 7ff8b6ffd700 1 -- 172.21.15.195:0/3592239420 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8b00095f0 con 0x7ff8c00f5410 2023-10-12T22:36:07.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.471+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3592239420 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c0045ad0 con 0x7ff8c00f5410 2023-10-12T22:36:07.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.471+0000 7ff8b6ffd700 1 --2- 172.21.15.195:0/3592239420 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff8ac02ff60 0x7ff8ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:07.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.471+0000 7ff8b6ffd700 1 -- 172.21.15.195:0/3592239420 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8b0040fe0 con 0x7ff8c00f5410 2023-10-12T22:36:07.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.471+0000 7ff8bdd9b700 1 --2- 172.21.15.195:0/3592239420 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff8ac02ff60 0x7ff8ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:07.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.472+0000 7ff8bdd9b700 1 --2- 172.21.15.195:0/3592239420 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff8ac02ff60 0x7ff8ac032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7ff8a80096a0 tx=0x7ff8a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:07.478 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.476+0000 7ff8b6ffd700 1 -- 172.21.15.195:0/3592239420 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8b001d7f0 con 0x7ff8c00f5410 2023-10-12T22:36:07.682 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.681+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3592239420 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8c0108150 con 0x7ff8c00f5410 2023-10-12T22:36:07.683 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.682+0000 7ff8b6ffd700 1 -- 172.21.15.195:0/3592239420 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff8b0010230 con 0x7ff8c00f5410 2023-10-12T22:36:07.683 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:07.684 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:07.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.684+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3592239420 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff8ac02ff60 msgr2=0x7ff8ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:07.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.684+0000 7ff8c4ff8700 1 --2- 172.21.15.195:0/3592239420 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff8ac02ff60 0x7ff8ac032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7ff8a80096a0 tx=0x7ff8a8006b40).stop 2023-10-12T22:36:07.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.684+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3592239420 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8c00f5410 msgr2=0x7ff8c0106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:07.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.684+0000 7ff8c4ff8700 1 --2- 172.21.15.195:0/3592239420 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8c00f5410 0x7ff8c0106720 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff8b0013510 tx=0x7ff8b000b040).stop 2023-10-12T22:36:07.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.684+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3592239420 shutdown_connections 2023-10-12T22:36:07.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.684+0000 7ff8c4ff8700 1 --2- 172.21.15.195:0/3592239420 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff8ac02ff60 0x7ff8ac032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:07.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.684+0000 7ff8c4ff8700 1 --2- 172.21.15.195:0/3592239420 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8c00f5410 0x7ff8c0106720 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:07.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.684+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3592239420 >> 172.21.15.195:0/3592239420 conn(0x7ff8c00f0b70 msgr2=0x7ff8c005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:07.687 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.684+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3592239420 shutdown_connections 2023-10-12T22:36:07.687 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:07.684+0000 7ff8c4ff8700 1 -- 172.21.15.195:0/3592239420 wait complete. 2023-10-12T22:36:07.687 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:08.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:06.841380+0000 mgr.smithi131.efktma (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:08.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:07.683392+0000 mon.smithi131 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.195:0/3592239420' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:09.074 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:09.075 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:10.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.258+0000 7ffa41bd6700 1 -- 172.21.15.195:0/3022469836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa3c0601e0 msgr2=0x7ffa3c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:10.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.258+0000 7ffa41bd6700 1 --2- 172.21.15.195:0/3022469836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa3c0601e0 0x7ffa3c0f9190 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7ffa2c009fb0 tx=0x7ffa2c009b30).stop 2023-10-12T22:36:10.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.259+0000 7ffa41bd6700 1 -- 172.21.15.195:0/3022469836 shutdown_connections 2023-10-12T22:36:10.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.259+0000 7ffa41bd6700 1 --2- 172.21.15.195:0/3022469836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa3c0601e0 0x7ffa3c0f9190 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:10.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.259+0000 7ffa41bd6700 1 -- 172.21.15.195:0/3022469836 >> 172.21.15.195:0/3022469836 conn(0x7ffa3c0f0b70 msgr2=0x7ffa3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:10.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.259+0000 7ffa41bd6700 1 -- 172.21.15.195:0/3022469836 shutdown_connections 2023-10-12T22:36:10.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.259+0000 7ffa41bd6700 1 -- 172.21.15.195:0/3022469836 wait complete. 2023-10-12T22:36:10.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.259+0000 7ffa41bd6700 1 Processor -- start 2023-10-12T22:36:10.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.259+0000 7ffa41bd6700 1 -- start start 2023-10-12T22:36:10.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.259+0000 7ffa41bd6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa3c0601e0 0x7ffa3c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:10.263 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.259+0000 7ffa41bd6700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c106bd0 con 0x7ffa3c0601e0 2023-10-12T22:36:10.264 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.260+0000 7ffa3b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa3c0601e0 0x7ffa3c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:10.264 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.260+0000 7ffa3b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa3c0601e0 0x7ffa3c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:39056/0 (socket says 172.21.15.195:39056) 2023-10-12T22:36:10.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.260+0000 7ffa3b7fe700 1 -- 172.21.15.195:0/2599972770 learned_addr learned my addr 172.21.15.195:0/2599972770 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:10.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.260+0000 7ffa3b7fe700 1 -- 172.21.15.195:0/2599972770 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c005040 con 0x7ffa3c0601e0 2023-10-12T22:36:10.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.260+0000 7ffa3b7fe700 1 --2- 172.21.15.195:0/2599972770 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa3c0601e0 0x7ffa3c1066e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7ffa2c004dd0 tx=0x7ffa2c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:10.265 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.260+0000 7ffa33fff700 1 -- 172.21.15.195:0/2599972770 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa2c018070 con 0x7ffa3c0601e0 2023-10-12T22:36:10.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.260+0000 7ffa41bd6700 1 -- 172.21.15.195:0/2599972770 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa3c106dd0 con 0x7ffa3c0601e0 2023-10-12T22:36:10.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.260+0000 7ffa33fff700 1 -- 172.21.15.195:0/2599972770 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa2c013410 con 0x7ffa3c0601e0 2023-10-12T22:36:10.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.260+0000 7ffa33fff700 1 -- 172.21.15.195:0/2599972770 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa2c01c3c0 con 0x7ffa3c0601e0 2023-10-12T22:36:10.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.260+0000 7ffa41bd6700 1 -- 172.21.15.195:0/2599972770 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa3c107260 con 0x7ffa3c0601e0 2023-10-12T22:36:10.266 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.261+0000 7ffa33fff700 1 -- 172.21.15.195:0/2599972770 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffa2c01d420 con 0x7ffa3c0601e0 2023-10-12T22:36:10.267 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.261+0000 7ffa41bd6700 1 -- 172.21.15.195:0/2599972770 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa3c045ad0 con 0x7ffa3c0601e0 2023-10-12T22:36:10.267 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.261+0000 7ffa33fff700 1 --2- 172.21.15.195:0/2599972770 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffa2802ff60 0x7ffa28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:10.267 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.261+0000 7ffa33fff700 1 -- 172.21.15.195:0/2599972770 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa2c041310 con 0x7ffa3c0601e0 2023-10-12T22:36:10.267 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.262+0000 7ffa3affd700 1 --2- 172.21.15.195:0/2599972770 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffa2802ff60 0x7ffa28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:10.267 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.262+0000 7ffa3affd700 1 --2- 172.21.15.195:0/2599972770 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffa2802ff60 0x7ffa28032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ffa240096a0 tx=0x7ffa24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:10.268 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.266+0000 7ffa33fff700 1 -- 172.21.15.195:0/2599972770 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa2c01b940 con 0x7ffa3c0601e0 2023-10-12T22:36:10.472 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.471+0000 7ffa41bd6700 1 -- 172.21.15.195:0/2599972770 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa3c1080b0 con 0x7ffa3c0601e0 2023-10-12T22:36:10.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.472+0000 7ffa33fff700 1 -- 172.21.15.195:0/2599972770 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffa2c01b370 con 0x7ffa3c0601e0 2023-10-12T22:36:10.474 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:10.474 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:10.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.474+0000 7ffa41bd6700 1 -- 172.21.15.195:0/2599972770 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffa2802ff60 msgr2=0x7ffa28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:10.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.474+0000 7ffa41bd6700 1 --2- 172.21.15.195:0/2599972770 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffa2802ff60 0x7ffa28032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ffa240096a0 tx=0x7ffa24006b40).stop 2023-10-12T22:36:10.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.474+0000 7ffa41bd6700 1 -- 172.21.15.195:0/2599972770 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa3c0601e0 msgr2=0x7ffa3c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:10.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.474+0000 7ffa41bd6700 1 --2- 172.21.15.195:0/2599972770 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa3c0601e0 0x7ffa3c1066e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7ffa2c004dd0 tx=0x7ffa2c009b30).stop 2023-10-12T22:36:10.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.474+0000 7ffa41bd6700 1 -- 172.21.15.195:0/2599972770 shutdown_connections 2023-10-12T22:36:10.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.474+0000 7ffa41bd6700 1 --2- 172.21.15.195:0/2599972770 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffa2802ff60 0x7ffa28032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:10.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.474+0000 7ffa41bd6700 1 --2- 172.21.15.195:0/2599972770 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa3c0601e0 0x7ffa3c1066e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:10.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.474+0000 7ffa41bd6700 1 -- 172.21.15.195:0/2599972770 >> 172.21.15.195:0/2599972770 conn(0x7ffa3c0f0b70 msgr2=0x7ffa3c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:10.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.474+0000 7ffa41bd6700 1 -- 172.21.15.195:0/2599972770 shutdown_connections 2023-10-12T22:36:10.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:10.474+0000 7ffa41bd6700 1 -- 172.21.15.195:0/2599972770 wait complete. 2023-10-12T22:36:10.477 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:10.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:08.841689+0000 mgr.smithi131.efktma (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:11.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:10.473395+0000 mon.smithi131 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.195:0/2599972770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:11.895 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:11.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:12.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:10.841966+0000 mgr.smithi131.efktma (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:13.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.083+0000 7ff467180700 1 -- 172.21.15.195:0/1551835094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4600f51a0 msgr2=0x7ff4600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:13.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.083+0000 7ff467180700 1 --2- 172.21.15.195:0/1551835094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4600f51a0 0x7ff4600f55c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7ff45c009fb0 tx=0x7ff45c009b30).stop 2023-10-12T22:36:13.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.083+0000 7ff467180700 1 -- 172.21.15.195:0/1551835094 shutdown_connections 2023-10-12T22:36:13.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.083+0000 7ff467180700 1 --2- 172.21.15.195:0/1551835094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4600f51a0 0x7ff4600f55c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:13.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.083+0000 7ff467180700 1 -- 172.21.15.195:0/1551835094 >> 172.21.15.195:0/1551835094 conn(0x7ff4600f09a0 msgr2=0x7ff4600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:13.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.083+0000 7ff467180700 1 -- 172.21.15.195:0/1551835094 shutdown_connections 2023-10-12T22:36:13.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.083+0000 7ff467180700 1 -- 172.21.15.195:0/1551835094 wait complete. 2023-10-12T22:36:13.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.084+0000 7ff467180700 1 Processor -- start 2023-10-12T22:36:13.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.084+0000 7ff467180700 1 -- start start 2023-10-12T22:36:13.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.084+0000 7ff467180700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4600f51a0 0x7ff4601091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:13.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.084+0000 7ff467180700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff460060220 con 0x7ff4600f51a0 2023-10-12T22:36:13.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.084+0000 7ff46617e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4600f51a0 0x7ff4601091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:13.090 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.084+0000 7ff46617e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4600f51a0 0x7ff4601091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:39084/0 (socket says 172.21.15.195:39084) 2023-10-12T22:36:13.090 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.084+0000 7ff46617e700 1 -- 172.21.15.195:0/1897884057 learned_addr learned my addr 172.21.15.195:0/1897884057 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:13.090 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.085+0000 7ff46617e700 1 -- 172.21.15.195:0/1897884057 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c005040 con 0x7ff4600f51a0 2023-10-12T22:36:13.090 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.085+0000 7ff46617e700 1 --2- 172.21.15.195:0/1897884057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4600f51a0 0x7ff4601091d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7ff45c013510 tx=0x7ff45c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:13.090 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.085+0000 7ff456ffd700 1 -- 172.21.15.195:0/1897884057 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff45c009320 con 0x7ff4600f51a0 2023-10-12T22:36:13.091 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.085+0000 7ff467180700 1 -- 172.21.15.195:0/1897884057 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff460060420 con 0x7ff4600f51a0 2023-10-12T22:36:13.091 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.085+0000 7ff456ffd700 1 -- 172.21.15.195:0/1897884057 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff45c013f00 con 0x7ff4600f51a0 2023-10-12T22:36:13.091 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.085+0000 7ff456ffd700 1 -- 172.21.15.195:0/1897884057 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff45c014dc0 con 0x7ff4600f51a0 2023-10-12T22:36:13.091 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.085+0000 7ff467180700 1 -- 172.21.15.195:0/1897884057 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff460109880 con 0x7ff4600f51a0 2023-10-12T22:36:13.091 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.086+0000 7ff456ffd700 1 -- 172.21.15.195:0/1897884057 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff45c014420 con 0x7ff4600f51a0 2023-10-12T22:36:13.092 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.086+0000 7ff467180700 1 -- 172.21.15.195:0/1897884057 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff460059100 con 0x7ff4600f51a0 2023-10-12T22:36:13.092 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.086+0000 7ff456ffd700 1 --2- 172.21.15.195:0/1897884057 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff44c02ff60 0x7ff44c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:13.092 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.086+0000 7ff456ffd700 1 -- 172.21.15.195:0/1897884057 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff45c041110 con 0x7ff4600f51a0 2023-10-12T22:36:13.093 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.086+0000 7ff46597d700 1 --2- 172.21.15.195:0/1897884057 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff44c02ff60 0x7ff44c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:13.093 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.087+0000 7ff46597d700 1 --2- 172.21.15.195:0/1897884057 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff44c02ff60 0x7ff44c032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff4500096a0 tx=0x7ff450006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:13.093 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.091+0000 7ff456ffd700 1 -- 172.21.15.195:0/1897884057 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff45c0095f0 con 0x7ff4600f51a0 2023-10-12T22:36:13.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.299+0000 7ff467180700 1 -- 172.21.15.195:0/1897884057 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff460045ad0 con 0x7ff4600f51a0 2023-10-12T22:36:13.301 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.300+0000 7ff456ffd700 1 -- 172.21.15.195:0/1897884057 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff45c01cb10 con 0x7ff4600f51a0 2023-10-12T22:36:13.301 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:13.301 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:13.303 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.301+0000 7ff467180700 1 -- 172.21.15.195:0/1897884057 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff44c02ff60 msgr2=0x7ff44c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:13.303 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.301+0000 7ff467180700 1 --2- 172.21.15.195:0/1897884057 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff44c02ff60 0x7ff44c032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff4500096a0 tx=0x7ff450006b40).stop 2023-10-12T22:36:13.303 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.301+0000 7ff467180700 1 -- 172.21.15.195:0/1897884057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4600f51a0 msgr2=0x7ff4601091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:13.303 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.301+0000 7ff467180700 1 --2- 172.21.15.195:0/1897884057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4600f51a0 0x7ff4601091d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7ff45c013510 tx=0x7ff45c00b040).stop 2023-10-12T22:36:13.303 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.301+0000 7ff467180700 1 -- 172.21.15.195:0/1897884057 shutdown_connections 2023-10-12T22:36:13.304 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.301+0000 7ff467180700 1 --2- 172.21.15.195:0/1897884057 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff44c02ff60 0x7ff44c032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:13.304 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.301+0000 7ff467180700 1 --2- 172.21.15.195:0/1897884057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4600f51a0 0x7ff4601091d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:13.304 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.301+0000 7ff467180700 1 -- 172.21.15.195:0/1897884057 >> 172.21.15.195:0/1897884057 conn(0x7ff4600f09a0 msgr2=0x7ff4600f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:13.304 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.301+0000 7ff467180700 1 -- 172.21.15.195:0/1897884057 shutdown_connections 2023-10-12T22:36:13.304 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:13.301+0000 7ff467180700 1 -- 172.21.15.195:0/1897884057 wait complete. 2023-10-12T22:36:13.305 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:14.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:14 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:12.842238+0000 mgr.smithi131.efktma (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:14.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:14 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:13.300959+0000 mon.smithi131 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.195:0/1897884057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:14.630 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:14.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:15.797 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.794+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/3046690379 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d28060280 msgr2=0x7f9d280f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:15.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.794+0000 7f9d2d9e1700 1 --2- 172.21.15.195:0/3046690379 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d28060280 0x7f9d280f90f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f9d1c009fb0 tx=0x7f9d1c009b30).stop 2023-10-12T22:36:15.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.794+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/3046690379 shutdown_connections 2023-10-12T22:36:15.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.794+0000 7f9d2d9e1700 1 --2- 172.21.15.195:0/3046690379 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d28060280 0x7f9d280f90f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:15.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.794+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/3046690379 >> 172.21.15.195:0/3046690379 conn(0x7f9d280f0b70 msgr2=0x7f9d280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:15.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.794+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/3046690379 shutdown_connections 2023-10-12T22:36:15.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.795+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/3046690379 wait complete. 2023-10-12T22:36:15.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.795+0000 7f9d2d9e1700 1 Processor -- start 2023-10-12T22:36:15.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.795+0000 7f9d2d9e1700 1 -- start start 2023-10-12T22:36:15.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.795+0000 7f9d2d9e1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d28060280 0x7f9d281066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:15.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.795+0000 7f9d2d9e1700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d28106bd0 con 0x7f9d28060280 2023-10-12T22:36:15.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.796+0000 7f9d26ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d28060280 0x7f9d281066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:15.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.796+0000 7f9d26ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d28060280 0x7f9d281066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:39108/0 (socket says 172.21.15.195:39108) 2023-10-12T22:36:15.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.796+0000 7f9d26ffd700 1 -- 172.21.15.195:0/1877605918 learned_addr learned my addr 172.21.15.195:0/1877605918 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:15.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.796+0000 7f9d26ffd700 1 -- 172.21.15.195:0/1877605918 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c005040 con 0x7f9d28060280 2023-10-12T22:36:15.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.796+0000 7f9d26ffd700 1 --2- 172.21.15.195:0/1877605918 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d28060280 0x7f9d281066e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f9d1c009830 tx=0x7f9d1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:15.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.796+0000 7f9d0f7fe700 1 -- 172.21.15.195:0/1877605918 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d1c009320 con 0x7f9d28060280 2023-10-12T22:36:15.801 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.796+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/1877605918 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d28106dd0 con 0x7f9d28060280 2023-10-12T22:36:15.801 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.796+0000 7f9d0f7fe700 1 -- 172.21.15.195:0/1877605918 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d1c0136c0 con 0x7f9d28060280 2023-10-12T22:36:15.801 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.796+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/1877605918 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d28107260 con 0x7f9d28060280 2023-10-12T22:36:15.801 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.796+0000 7f9d0f7fe700 1 -- 172.21.15.195:0/1877605918 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d1c01ee90 con 0x7f9d28060280 2023-10-12T22:36:15.802 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.797+0000 7f9d0f7fe700 1 -- 172.21.15.195:0/1877605918 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d1c014420 con 0x7f9d28060280 2023-10-12T22:36:15.803 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.797+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/1877605918 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d28045ad0 con 0x7f9d28060280 2023-10-12T22:36:15.803 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.797+0000 7f9d0f7fe700 1 --2- 172.21.15.195:0/1877605918 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9d1002ff10 0x7f9d100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:15.803 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.797+0000 7f9d0f7fe700 1 -- 172.21.15.195:0/1877605918 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d1c040f40 con 0x7f9d28060280 2023-10-12T22:36:15.803 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.797+0000 7f9d267fc700 1 --2- 172.21.15.195:0/1877605918 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9d1002ff10 0x7f9d100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:15.803 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.798+0000 7f9d267fc700 1 --2- 172.21.15.195:0/1877605918 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9d1002ff10 0x7f9d100323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9d180096a0 tx=0x7f9d18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:15.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:15.803+0000 7f9d0f7fe700 1 -- 172.21.15.195:0/1877605918 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d1c01c6a0 con 0x7f9d28060280 2023-10-12T22:36:16.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:16.010+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/1877605918 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d28108110 con 0x7f9d28060280 2023-10-12T22:36:16.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:16.011+0000 7f9d0f7fe700 1 -- 172.21.15.195:0/1877605918 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d1c02c2d0 con 0x7f9d28060280 2023-10-12T22:36:16.013 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:16.013 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:16.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:16.013+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/1877605918 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9d1002ff10 msgr2=0x7f9d100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:16.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:16.013+0000 7f9d2d9e1700 1 --2- 172.21.15.195:0/1877605918 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9d1002ff10 0x7f9d100323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9d180096a0 tx=0x7f9d18006b40).stop 2023-10-12T22:36:16.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:16.013+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/1877605918 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d28060280 msgr2=0x7f9d281066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:16.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:16.013+0000 7f9d2d9e1700 1 --2- 172.21.15.195:0/1877605918 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d28060280 0x7f9d281066e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f9d1c009830 tx=0x7f9d1c00b040).stop 2023-10-12T22:36:16.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:16.013+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/1877605918 shutdown_connections 2023-10-12T22:36:16.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:16.013+0000 7f9d2d9e1700 1 --2- 172.21.15.195:0/1877605918 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9d1002ff10 0x7f9d100323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:16.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:16.013+0000 7f9d2d9e1700 1 --2- 172.21.15.195:0/1877605918 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d28060280 0x7f9d281066e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:16.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:16.013+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/1877605918 >> 172.21.15.195:0/1877605918 conn(0x7f9d280f0b70 msgr2=0x7f9d280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:16.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:16.013+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/1877605918 shutdown_connections 2023-10-12T22:36:16.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:16.013+0000 7f9d2d9e1700 1 -- 172.21.15.195:0/1877605918 wait complete. 2023-10-12T22:36:16.017 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:16.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:16 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:14.842517+0000 mgr.smithi131.efktma (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:16.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:16 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:16.012517+0000 mon.smithi131 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.195:0/1877605918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:17.392 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:17.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:18.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:18 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:16.842762+0000 mgr.smithi131.efktma (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:18.603 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.602+0000 7f3f4f27e700 1 -- 172.21.15.195:0/1827712822 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3f480f51a0 msgr2=0x7f3f480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:18.603 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.602+0000 7f3f4f27e700 1 --2- 172.21.15.195:0/1827712822 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3f480f51a0 0x7f3f480f55c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f3f38009fb0 tx=0x7f3f38009b30).stop 2023-10-12T22:36:18.603 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.602+0000 7f3f4f27e700 1 -- 172.21.15.195:0/1827712822 shutdown_connections 2023-10-12T22:36:18.603 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.602+0000 7f3f4f27e700 1 --2- 172.21.15.195:0/1827712822 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3f480f51a0 0x7f3f480f55c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:18.604 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.602+0000 7f3f4f27e700 1 -- 172.21.15.195:0/1827712822 >> 172.21.15.195:0/1827712822 conn(0x7f3f480f09a0 msgr2=0x7f3f480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:18.604 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.602+0000 7f3f4f27e700 1 -- 172.21.15.195:0/1827712822 shutdown_connections 2023-10-12T22:36:18.604 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.602+0000 7f3f4f27e700 1 -- 172.21.15.195:0/1827712822 wait complete. 2023-10-12T22:36:18.604 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.602+0000 7f3f4f27e700 1 Processor -- start 2023-10-12T22:36:18.604 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.603+0000 7f3f4f27e700 1 -- start start 2023-10-12T22:36:18.605 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.603+0000 7f3f4f27e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3f480f51a0 0x7f3f480fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:18.605 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.603+0000 7f3f4f27e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f480fb5a0 con 0x7f3f480f51a0 2023-10-12T22:36:18.607 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.603+0000 7f3f4e27c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3f480f51a0 0x7f3f480fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:18.607 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.603+0000 7f3f4e27c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3f480f51a0 0x7f3f480fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:59046/0 (socket says 172.21.15.195:59046) 2023-10-12T22:36:18.607 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.603+0000 7f3f4e27c700 1 -- 172.21.15.195:0/3179322594 learned_addr learned my addr 172.21.15.195:0/3179322594 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:18.607 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.603+0000 7f3f4e27c700 1 -- 172.21.15.195:0/3179322594 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f38005040 con 0x7f3f480f51a0 2023-10-12T22:36:18.607 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.603+0000 7f3f4e27c700 1 --2- 172.21.15.195:0/3179322594 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3f480f51a0 0x7f3f480fd120 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f3f38013510 tx=0x7f3f3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:18.608 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.604+0000 7f3f3effd700 1 -- 172.21.15.195:0/3179322594 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f38009320 con 0x7f3f480f51a0 2023-10-12T22:36:18.608 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.604+0000 7f3f4f27e700 1 -- 172.21.15.195:0/3179322594 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f480fb740 con 0x7f3f480f51a0 2023-10-12T22:36:18.608 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.604+0000 7f3f3effd700 1 -- 172.21.15.195:0/3179322594 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f38013f00 con 0x7f3f480f51a0 2023-10-12T22:36:18.608 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.604+0000 7f3f3effd700 1 -- 172.21.15.195:0/3179322594 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f38014dc0 con 0x7f3f480f51a0 2023-10-12T22:36:18.608 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.604+0000 7f3f4f27e700 1 -- 172.21.15.195:0/3179322594 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f480fbbd0 con 0x7f3f480f51a0 2023-10-12T22:36:18.609 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.604+0000 7f3f3effd700 1 -- 172.21.15.195:0/3179322594 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f38014420 con 0x7f3f480f51a0 2023-10-12T22:36:18.609 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.604+0000 7f3f4f27e700 1 -- 172.21.15.195:0/3179322594 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f48059100 con 0x7f3f480f51a0 2023-10-12T22:36:18.609 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.605+0000 7f3f3effd700 1 --2- 172.21.15.195:0/3179322594 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3f3402ff60 0x7f3f34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:18.609 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.605+0000 7f3f3effd700 1 -- 172.21.15.195:0/3179322594 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f38041020 con 0x7f3f480f51a0 2023-10-12T22:36:18.609 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.605+0000 7f3f4da7b700 1 --2- 172.21.15.195:0/3179322594 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3f3402ff60 0x7f3f34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:18.610 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.605+0000 7f3f4da7b700 1 --2- 172.21.15.195:0/3179322594 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3f3402ff60 0x7f3f34032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f3f440096a0 tx=0x7f3f44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:18.610 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.609+0000 7f3f3effd700 1 -- 172.21.15.195:0/3179322594 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f380095f0 con 0x7f3f480f51a0 2023-10-12T22:36:18.816 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.816+0000 7f3f4f27e700 1 -- 172.21.15.195:0/3179322594 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f48045ad0 con 0x7f3f480f51a0 2023-10-12T22:36:18.817 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.816+0000 7f3f3effd700 1 -- 172.21.15.195:0/3179322594 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3f3801cb10 con 0x7f3f480f51a0 2023-10-12T22:36:18.817 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:18.817 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:18.819 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.818+0000 7f3f4f27e700 1 -- 172.21.15.195:0/3179322594 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3f3402ff60 msgr2=0x7f3f34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:18.819 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.818+0000 7f3f4f27e700 1 --2- 172.21.15.195:0/3179322594 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3f3402ff60 0x7f3f34032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f3f440096a0 tx=0x7f3f44006b40).stop 2023-10-12T22:36:18.819 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.818+0000 7f3f4f27e700 1 -- 172.21.15.195:0/3179322594 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3f480f51a0 msgr2=0x7f3f480fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:18.819 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.818+0000 7f3f4f27e700 1 --2- 172.21.15.195:0/3179322594 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3f480f51a0 0x7f3f480fd120 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f3f38013510 tx=0x7f3f3800b040).stop 2023-10-12T22:36:18.821 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.818+0000 7f3f4f27e700 1 -- 172.21.15.195:0/3179322594 shutdown_connections 2023-10-12T22:36:18.821 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.818+0000 7f3f4f27e700 1 --2- 172.21.15.195:0/3179322594 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3f3402ff60 0x7f3f34032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:18.822 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.818+0000 7f3f4f27e700 1 --2- 172.21.15.195:0/3179322594 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3f480f51a0 0x7f3f480fd120 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:18.822 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.818+0000 7f3f4f27e700 1 -- 172.21.15.195:0/3179322594 >> 172.21.15.195:0/3179322594 conn(0x7f3f480f09a0 msgr2=0x7f3f4805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:18.822 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.818+0000 7f3f4f27e700 1 -- 172.21.15.195:0/3179322594 shutdown_connections 2023-10-12T22:36:18.822 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:18.818+0000 7f3f4f27e700 1 -- 172.21.15.195:0/3179322594 wait complete. 2023-10-12T22:36:18.822 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:19.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:19 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:18.816770+0000 mon.smithi131 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.195:0/3179322594' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:20.201 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:20.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:20.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:20 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:18.843112+0000 mgr.smithi131.efktma (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:21.342 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.342+0000 7fced8b47700 1 -- 172.21.15.195:0/1882888460 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fced40f62a0 msgr2=0x7fced40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:21.343 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.342+0000 7fced8b47700 1 --2- 172.21.15.195:0/1882888460 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fced40f62a0 0x7fced40f66c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fcec4009fb0 tx=0x7fcec4009b30).stop 2023-10-12T22:36:21.343 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.342+0000 7fced8b47700 1 -- 172.21.15.195:0/1882888460 shutdown_connections 2023-10-12T22:36:21.343 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.342+0000 7fced8b47700 1 --2- 172.21.15.195:0/1882888460 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fced40f62a0 0x7fced40f66c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:21.343 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.342+0000 7fced8b47700 1 -- 172.21.15.195:0/1882888460 >> 172.21.15.195:0/1882888460 conn(0x7fced40f1aa0 msgr2=0x7fced40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:21.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.342+0000 7fced8b47700 1 -- 172.21.15.195:0/1882888460 shutdown_connections 2023-10-12T22:36:21.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.342+0000 7fced8b47700 1 -- 172.21.15.195:0/1882888460 wait complete. 2023-10-12T22:36:21.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.342+0000 7fced8b47700 1 Processor -- start 2023-10-12T22:36:21.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.342+0000 7fced8b47700 1 -- start start 2023-10-12T22:36:21.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.343+0000 7fced8b47700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fced40f62a0 0x7fced41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:21.345 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.343+0000 7fced8b47700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced41069a0 con 0x7fced40f62a0 2023-10-12T22:36:21.346 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.343+0000 7fced37fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fced40f62a0 0x7fced41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:21.346 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.343+0000 7fced37fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fced40f62a0 0x7fced41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:59062/0 (socket says 172.21.15.195:59062) 2023-10-12T22:36:21.347 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.343+0000 7fced37fe700 1 -- 172.21.15.195:0/1498048779 learned_addr learned my addr 172.21.15.195:0/1498048779 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:21.347 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.343+0000 7fced37fe700 1 -- 172.21.15.195:0/1498048779 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec4005040 con 0x7fced40f62a0 2023-10-12T22:36:21.347 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.343+0000 7fced37fe700 1 --2- 172.21.15.195:0/1498048779 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fced40f62a0 0x7fced41064b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fcec4013510 tx=0x7fcec400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:21.347 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.343+0000 7fcebbfff700 1 -- 172.21.15.195:0/1498048779 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcec4009320 con 0x7fced40f62a0 2023-10-12T22:36:21.347 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.343+0000 7fced8b47700 1 -- 172.21.15.195:0/1498048779 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced4106ba0 con 0x7fced40f62a0 2023-10-12T22:36:21.348 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.343+0000 7fced8b47700 1 -- 172.21.15.195:0/1498048779 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced4107030 con 0x7fced40f62a0 2023-10-12T22:36:21.348 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.344+0000 7fcebbfff700 1 -- 172.21.15.195:0/1498048779 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcec4013f00 con 0x7fced40f62a0 2023-10-12T22:36:21.348 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.344+0000 7fcebbfff700 1 -- 172.21.15.195:0/1498048779 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcec4014d90 con 0x7fced40f62a0 2023-10-12T22:36:21.348 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.344+0000 7fcebbfff700 1 -- 172.21.15.195:0/1498048779 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcec4014420 con 0x7fced40f62a0 2023-10-12T22:36:21.348 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.344+0000 7fcebbfff700 1 --2- 172.21.15.195:0/1498048779 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcebc02ff60 0x7fcebc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:21.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.344+0000 7fced8b47700 1 -- 172.21.15.195:0/1498048779 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced4045ad0 con 0x7fced40f62a0 2023-10-12T22:36:21.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.344+0000 7fcebbfff700 1 -- 172.21.15.195:0/1498048779 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcec40413a0 con 0x7fced40f62a0 2023-10-12T22:36:21.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.345+0000 7fced2ffd700 1 --2- 172.21.15.195:0/1498048779 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcebc02ff60 0x7fcebc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:21.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.345+0000 7fced2ffd700 1 --2- 172.21.15.195:0/1498048779 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcebc02ff60 0x7fcebc032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fcec80096a0 tx=0x7fcec8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:21.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.349+0000 7fcebbfff700 1 -- 172.21.15.195:0/1498048779 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcec401c320 con 0x7fced40f62a0 2023-10-12T22:36:21.560 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.560+0000 7fced8b47700 1 -- 172.21.15.195:0/1498048779 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fced40fbce0 con 0x7fced40f62a0 2023-10-12T22:36:21.560 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.560+0000 7fcebbfff700 1 -- 172.21.15.195:0/1498048779 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcec402b030 con 0x7fced40f62a0 2023-10-12T22:36:21.561 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:21.561 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:21.562 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.562+0000 7fced8b47700 1 -- 172.21.15.195:0/1498048779 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcebc02ff60 msgr2=0x7fcebc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:21.562 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.562+0000 7fced8b47700 1 --2- 172.21.15.195:0/1498048779 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcebc02ff60 0x7fcebc032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fcec80096a0 tx=0x7fcec8006b40).stop 2023-10-12T22:36:21.563 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.562+0000 7fced8b47700 1 -- 172.21.15.195:0/1498048779 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fced40f62a0 msgr2=0x7fced41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:21.563 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.562+0000 7fced8b47700 1 --2- 172.21.15.195:0/1498048779 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fced40f62a0 0x7fced41064b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fcec4013510 tx=0x7fcec400b040).stop 2023-10-12T22:36:21.563 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.562+0000 7fced8b47700 1 -- 172.21.15.195:0/1498048779 shutdown_connections 2023-10-12T22:36:21.563 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.562+0000 7fced8b47700 1 --2- 172.21.15.195:0/1498048779 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcebc02ff60 0x7fcebc032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:21.563 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.562+0000 7fced8b47700 1 --2- 172.21.15.195:0/1498048779 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fced40f62a0 0x7fced41064b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:21.564 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.562+0000 7fced8b47700 1 -- 172.21.15.195:0/1498048779 >> 172.21.15.195:0/1498048779 conn(0x7fced40f1aa0 msgr2=0x7fced40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:21.564 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.562+0000 7fced8b47700 1 -- 172.21.15.195:0/1498048779 shutdown_connections 2023-10-12T22:36:21.564 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:21.562+0000 7fced8b47700 1 -- 172.21.15.195:0/1498048779 wait complete. 2023-10-12T22:36:21.564 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:22.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:20.843377+0000 mgr.smithi131.efktma (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:22.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:21.560812+0000 mon.smithi131 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.195:0/1498048779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:22.940 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:22.941 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:23.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:22.935041+0000 mon.smithi131 (mon.0) 222 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/mirror_snapshot_schedule"}]: dispatch 2023-10-12T22:36:23.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:22.941847+0000 mon.smithi131 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/trash_purge_schedule"}]: dispatch 2023-10-12T22:36:24.083 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.083+0000 7f8d21c9f700 1 -- 172.21.15.195:0/2080363567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1c0f6510 msgr2=0x7f8d1c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:24.083 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.083+0000 7f8d21c9f700 1 --2- 172.21.15.195:0/2080363567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1c0f6510 0x7f8d1c0f6930 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f8d14009fb0 tx=0x7f8d14009b30).stop 2023-10-12T22:36:24.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.083+0000 7f8d21c9f700 1 -- 172.21.15.195:0/2080363567 shutdown_connections 2023-10-12T22:36:24.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.083+0000 7f8d21c9f700 1 --2- 172.21.15.195:0/2080363567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1c0f6510 0x7f8d1c0f6930 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:24.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.083+0000 7f8d21c9f700 1 -- 172.21.15.195:0/2080363567 >> 172.21.15.195:0/2080363567 conn(0x7f8d1c0f1c70 msgr2=0x7f8d1c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:24.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.083+0000 7f8d21c9f700 1 -- 172.21.15.195:0/2080363567 shutdown_connections 2023-10-12T22:36:24.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.083+0000 7f8d21c9f700 1 -- 172.21.15.195:0/2080363567 wait complete. 2023-10-12T22:36:24.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.083+0000 7f8d21c9f700 1 Processor -- start 2023-10-12T22:36:24.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.084+0000 7f8d21c9f700 1 -- start start 2023-10-12T22:36:24.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.084+0000 7f8d21c9f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1c0f6510 0x7f8d1c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:24.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.084+0000 7f8d21c9f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c106c20 con 0x7f8d1c0f6510 2023-10-12T22:36:24.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.084+0000 7f8d1b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1c0f6510 0x7f8d1c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:24.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.084+0000 7f8d1b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1c0f6510 0x7f8d1c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:59070/0 (socket says 172.21.15.195:59070) 2023-10-12T22:36:24.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.084+0000 7f8d1b7fe700 1 -- 172.21.15.195:0/3876668621 learned_addr learned my addr 172.21.15.195:0/3876668621 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:24.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.085+0000 7f8d1b7fe700 1 -- 172.21.15.195:0/3876668621 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d14005040 con 0x7f8d1c0f6510 2023-10-12T22:36:24.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.085+0000 7f8d1b7fe700 1 --2- 172.21.15.195:0/3876668621 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1c0f6510 0x7f8d1c106730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f8d14013510 tx=0x7f8d1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:24.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.085+0000 7f8d03fff700 1 -- 172.21.15.195:0/3876668621 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d14009320 con 0x7f8d1c0f6510 2023-10-12T22:36:24.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.085+0000 7f8d21c9f700 1 -- 172.21.15.195:0/3876668621 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d1c106e20 con 0x7f8d1c0f6510 2023-10-12T22:36:24.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.085+0000 7f8d03fff700 1 -- 172.21.15.195:0/3876668621 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d14013f00 con 0x7f8d1c0f6510 2023-10-12T22:36:24.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.085+0000 7f8d03fff700 1 -- 172.21.15.195:0/3876668621 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d14014d90 con 0x7f8d1c0f6510 2023-10-12T22:36:24.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.085+0000 7f8d21c9f700 1 -- 172.21.15.195:0/3876668621 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1c1072b0 con 0x7f8d1c0f6510 2023-10-12T22:36:24.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.086+0000 7f8d03fff700 1 -- 172.21.15.195:0/3876668621 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d140095f0 con 0x7f8d1c0f6510 2023-10-12T22:36:24.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.086+0000 7f8d03fff700 1 --2- 172.21.15.195:0/3876668621 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d0402ff60 0x7f8d04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:24.090 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.086+0000 7f8d03fff700 1 -- 172.21.15.195:0/3876668621 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d14041000 con 0x7f8d1c0f6510 2023-10-12T22:36:24.090 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.086+0000 7f8d21c9f700 1 -- 172.21.15.195:0/3876668621 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d1c045ad0 con 0x7f8d1c0f6510 2023-10-12T22:36:24.090 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.086+0000 7f8d1affd700 1 --2- 172.21.15.195:0/3876668621 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d0402ff60 0x7f8d04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:24.090 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.086+0000 7f8d1affd700 1 --2- 172.21.15.195:0/3876668621 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d0402ff60 0x7f8d04032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8d100096a0 tx=0x7f8d10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:24.091 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.091+0000 7f8d03fff700 1 -- 172.21.15.195:0/3876668621 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d14014420 con 0x7f8d1c0f6510 2023-10-12T22:36:24.297 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.296+0000 7f8d21c9f700 1 -- 172.21.15.195:0/3876668621 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d1c0fc050 con 0x7f8d1c0f6510 2023-10-12T22:36:24.297 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.297+0000 7f8d03fff700 1 -- 172.21.15.195:0/3876668621 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8d1402b020 con 0x7f8d1c0f6510 2023-10-12T22:36:24.297 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:24.298 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:24.299 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.298+0000 7f8d21c9f700 1 -- 172.21.15.195:0/3876668621 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d0402ff60 msgr2=0x7f8d04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:24.299 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.298+0000 7f8d21c9f700 1 --2- 172.21.15.195:0/3876668621 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d0402ff60 0x7f8d04032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8d100096a0 tx=0x7f8d10006b40).stop 2023-10-12T22:36:24.299 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.298+0000 7f8d21c9f700 1 -- 172.21.15.195:0/3876668621 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1c0f6510 msgr2=0x7f8d1c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:24.299 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.298+0000 7f8d21c9f700 1 --2- 172.21.15.195:0/3876668621 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1c0f6510 0x7f8d1c106730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f8d14013510 tx=0x7f8d1400b040).stop 2023-10-12T22:36:24.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.298+0000 7f8d21c9f700 1 -- 172.21.15.195:0/3876668621 shutdown_connections 2023-10-12T22:36:24.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.298+0000 7f8d21c9f700 1 --2- 172.21.15.195:0/3876668621 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d0402ff60 0x7f8d04032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:24.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.298+0000 7f8d21c9f700 1 --2- 172.21.15.195:0/3876668621 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d1c0f6510 0x7f8d1c106730 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:24.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.298+0000 7f8d21c9f700 1 -- 172.21.15.195:0/3876668621 >> 172.21.15.195:0/3876668621 conn(0x7f8d1c0f1c70 msgr2=0x7f8d1c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:24.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.299+0000 7f8d21c9f700 1 -- 172.21.15.195:0/3876668621 shutdown_connections 2023-10-12T22:36:24.301 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:24.299+0000 7f8d21c9f700 1 -- 172.21.15.195:0/3876668621 wait complete. 2023-10-12T22:36:24.301 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:24.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:22.843701+0000 mgr.smithi131.efktma (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:25.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:25 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:24.297034+0000 mon.smithi131 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.195:0/3876668621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:25.689 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:25.690 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:26.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:26 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:24.844048+0000 mgr.smithi131.efktma (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:26.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.869+0000 7fb01d085700 1 -- 172.21.15.195:0/2210438617 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0180f6510 msgr2=0x7fb0180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:26.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.869+0000 7fb01d085700 1 --2- 172.21.15.195:0/2210438617 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0180f6510 0x7fb0180f6930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fb00c009fb0 tx=0x7fb00c009b30).stop 2023-10-12T22:36:26.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.869+0000 7fb01d085700 1 -- 172.21.15.195:0/2210438617 shutdown_connections 2023-10-12T22:36:26.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.869+0000 7fb01d085700 1 --2- 172.21.15.195:0/2210438617 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0180f6510 0x7fb0180f6930 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:26.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.869+0000 7fb01d085700 1 -- 172.21.15.195:0/2210438617 >> 172.21.15.195:0/2210438617 conn(0x7fb0180f1c70 msgr2=0x7fb0180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:26.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.869+0000 7fb01d085700 1 -- 172.21.15.195:0/2210438617 shutdown_connections 2023-10-12T22:36:26.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.869+0000 7fb01d085700 1 -- 172.21.15.195:0/2210438617 wait complete. 2023-10-12T22:36:26.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.869+0000 7fb01d085700 1 Processor -- start 2023-10-12T22:36:26.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.870+0000 7fb01d085700 1 -- start start 2023-10-12T22:36:26.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.870+0000 7fb01d085700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0180f6510 0x7fb018106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:26.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.870+0000 7fb01d085700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb018106c20 con 0x7fb0180f6510 2023-10-12T22:36:26.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.870+0000 7fb016d9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0180f6510 0x7fb018106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:26.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.870+0000 7fb016d9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0180f6510 0x7fb018106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:59086/0 (socket says 172.21.15.195:59086) 2023-10-12T22:36:26.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.870+0000 7fb016d9d700 1 -- 172.21.15.195:0/4047907270 learned_addr learned my addr 172.21.15.195:0/4047907270 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:26.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.870+0000 7fb016d9d700 1 -- 172.21.15.195:0/4047907270 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c005040 con 0x7fb0180f6510 2023-10-12T22:36:26.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.871+0000 7fb016d9d700 1 --2- 172.21.15.195:0/4047907270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0180f6510 0x7fb018106730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fb00c013510 tx=0x7fb00c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.871+0000 7fafff7fe700 1 -- 172.21.15.195:0/4047907270 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb00c009320 con 0x7fb0180f6510 2023-10-12T22:36:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.871+0000 7fb01d085700 1 -- 172.21.15.195:0/4047907270 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb018106e20 con 0x7fb0180f6510 2023-10-12T22:36:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.871+0000 7fafff7fe700 1 -- 172.21.15.195:0/4047907270 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb00c013f00 con 0x7fb0180f6510 2023-10-12T22:36:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.871+0000 7fafff7fe700 1 -- 172.21.15.195:0/4047907270 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb00c014d90 con 0x7fb0180f6510 2023-10-12T22:36:26.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.871+0000 7fb01d085700 1 -- 172.21.15.195:0/4047907270 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0181072b0 con 0x7fb0180f6510 2023-10-12T22:36:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.871+0000 7fafff7fe700 1 -- 172.21.15.195:0/4047907270 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb00c014420 con 0x7fb0180f6510 2023-10-12T22:36:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.872+0000 7fb01d085700 1 -- 172.21.15.195:0/4047907270 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb018045ad0 con 0x7fb0180f6510 2023-10-12T22:36:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.872+0000 7fafff7fe700 1 --2- 172.21.15.195:0/4047907270 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb00002ff60 0x7fb000032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.872+0000 7fafff7fe700 1 -- 172.21.15.195:0/4047907270 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb00c041000 con 0x7fb0180f6510 2023-10-12T22:36:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.872+0000 7fb01659c700 1 --2- 172.21.15.195:0/4047907270 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb00002ff60 0x7fb000032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:26.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.872+0000 7fb01659c700 1 --2- 172.21.15.195:0/4047907270 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb00002ff60 0x7fb000032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb0080096a0 tx=0x7fb008006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:26.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:26.877+0000 7fafff7fe700 1 -- 172.21.15.195:0/4047907270 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb00c0095f0 con 0x7fb0180f6510 2023-10-12T22:36:27.083 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:27.083+0000 7fb01d085700 1 -- 172.21.15.195:0/4047907270 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0180fbf60 con 0x7fb0180f6510 2023-10-12T22:36:27.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:27.084+0000 7fafff7fe700 1 -- 172.21.15.195:0/4047907270 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb00c01db70 con 0x7fb0180f6510 2023-10-12T22:36:27.085 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:27.085 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:27.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:27.085+0000 7fb01d085700 1 -- 172.21.15.195:0/4047907270 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb00002ff60 msgr2=0x7fb000032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:27.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:27.085+0000 7fb01d085700 1 --2- 172.21.15.195:0/4047907270 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb00002ff60 0x7fb000032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb0080096a0 tx=0x7fb008006b40).stop 2023-10-12T22:36:27.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:27.085+0000 7fb01d085700 1 -- 172.21.15.195:0/4047907270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0180f6510 msgr2=0x7fb018106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:27.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:27.085+0000 7fb01d085700 1 --2- 172.21.15.195:0/4047907270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0180f6510 0x7fb018106730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fb00c013510 tx=0x7fb00c00b040).stop 2023-10-12T22:36:27.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:27.085+0000 7fb01d085700 1 -- 172.21.15.195:0/4047907270 shutdown_connections 2023-10-12T22:36:27.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:27.085+0000 7fb01d085700 1 --2- 172.21.15.195:0/4047907270 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb00002ff60 0x7fb000032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:27.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:27.085+0000 7fb01d085700 1 --2- 172.21.15.195:0/4047907270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0180f6510 0x7fb018106730 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:27.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:27.085+0000 7fb01d085700 1 -- 172.21.15.195:0/4047907270 >> 172.21.15.195:0/4047907270 conn(0x7fb0180f1c70 msgr2=0x7fb0180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:27.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:27.085+0000 7fb01d085700 1 -- 172.21.15.195:0/4047907270 shutdown_connections 2023-10-12T22:36:27.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:27.085+0000 7fb01d085700 1 -- 172.21.15.195:0/4047907270 wait complete. 2023-10-12T22:36:27.088 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:28.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:28 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:26.844349+0000 mgr.smithi131.efktma (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:28.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:28 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:27.084269+0000 mon.smithi131 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.195:0/4047907270' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:28.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:28.491 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:29.585 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.583+0000 7f9eac6ca700 1 -- 172.21.15.195:0/459447246 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea40f6510 msgr2=0x7f9ea40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:29.585 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.583+0000 7f9eac6ca700 1 --2- 172.21.15.195:0/459447246 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea40f6510 0x7f9ea40f6930 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f9e9c009fb0 tx=0x7f9e9c009b30).stop 2023-10-12T22:36:29.585 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.583+0000 7f9eac6ca700 1 -- 172.21.15.195:0/459447246 shutdown_connections 2023-10-12T22:36:29.585 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.583+0000 7f9eac6ca700 1 --2- 172.21.15.195:0/459447246 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea40f6510 0x7f9ea40f6930 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:29.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.583+0000 7f9eac6ca700 1 -- 172.21.15.195:0/459447246 >> 172.21.15.195:0/459447246 conn(0x7f9ea40f1c70 msgr2=0x7f9ea40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:29.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.583+0000 7f9eac6ca700 1 -- 172.21.15.195:0/459447246 shutdown_connections 2023-10-12T22:36:29.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.583+0000 7f9eac6ca700 1 -- 172.21.15.195:0/459447246 wait complete. 2023-10-12T22:36:29.588 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.584+0000 7f9eac6ca700 1 Processor -- start 2023-10-12T22:36:29.589 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.584+0000 7f9eac6ca700 1 -- start start 2023-10-12T22:36:29.589 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.584+0000 7f9eac6ca700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea40f6510 0x7f9ea406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:29.589 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.584+0000 7f9eac6ca700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea4069e20 con 0x7f9ea40f6510 2023-10-12T22:36:29.589 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.584+0000 7f9eaa466700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea40f6510 0x7f9ea406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:29.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.584+0000 7f9eaa466700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea40f6510 0x7f9ea406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38594/0 (socket says 172.21.15.195:38594) 2023-10-12T22:36:29.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.584+0000 7f9eaa466700 1 -- 172.21.15.195:0/3317868240 learned_addr learned my addr 172.21.15.195:0/3317868240 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:29.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.585+0000 7f9eaa466700 1 -- 172.21.15.195:0/3317868240 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c005040 con 0x7f9ea40f6510 2023-10-12T22:36:29.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.585+0000 7f9eaa466700 1 --2- 172.21.15.195:0/3317868240 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea40f6510 0x7f9ea406b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f9e9c013510 tx=0x7f9e9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:29.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.585+0000 7f9ea2ffd700 1 -- 172.21.15.195:0/3317868240 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e9c009320 con 0x7f9ea40f6510 2023-10-12T22:36:29.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.585+0000 7f9eac6ca700 1 -- 172.21.15.195:0/3317868240 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea4069fc0 con 0x7f9ea40f6510 2023-10-12T22:36:29.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.585+0000 7f9ea2ffd700 1 -- 172.21.15.195:0/3317868240 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e9c013f00 con 0x7f9ea40f6510 2023-10-12T22:36:29.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.585+0000 7f9ea2ffd700 1 -- 172.21.15.195:0/3317868240 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e9c014d90 con 0x7f9ea40f6510 2023-10-12T22:36:29.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.585+0000 7f9eac6ca700 1 -- 172.21.15.195:0/3317868240 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea406a450 con 0x7f9ea40f6510 2023-10-12T22:36:29.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.586+0000 7f9eac6ca700 1 -- 172.21.15.195:0/3317868240 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea4059100 con 0x7f9ea40f6510 2023-10-12T22:36:29.592 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.586+0000 7f9ea2ffd700 1 -- 172.21.15.195:0/3317868240 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e9c0095f0 con 0x7f9ea40f6510 2023-10-12T22:36:29.592 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.586+0000 7f9ea2ffd700 1 --2- 172.21.15.195:0/3317868240 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9e8c02ff60 0x7f9e8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:29.592 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.586+0000 7f9ea2ffd700 1 -- 172.21.15.195:0/3317868240 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e9c041000 con 0x7f9ea40f6510 2023-10-12T22:36:29.592 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.586+0000 7f9ea9c65700 1 --2- 172.21.15.195:0/3317868240 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9e8c02ff60 0x7f9e8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:29.592 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.587+0000 7f9ea9c65700 1 --2- 172.21.15.195:0/3317868240 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9e8c02ff60 0x7f9e8c032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f9e980096a0 tx=0x7f9e98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.590+0000 7f9ea2ffd700 1 -- 172.21.15.195:0/3317868240 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e9c014420 con 0x7f9ea40f6510 2023-10-12T22:36:29.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.798+0000 7f9eac6ca700 1 -- 172.21.15.195:0/3317868240 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ea40fbf60 con 0x7f9ea40f6510 2023-10-12T22:36:29.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.799+0000 7f9ea2ffd700 1 -- 172.21.15.195:0/3317868240 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9e9c010230 con 0x7f9ea40f6510 2023-10-12T22:36:29.801 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:29.801 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:29.803 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.801+0000 7f9eac6ca700 1 -- 172.21.15.195:0/3317868240 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9e8c02ff60 msgr2=0x7f9e8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:29.803 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.801+0000 7f9eac6ca700 1 --2- 172.21.15.195:0/3317868240 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9e8c02ff60 0x7f9e8c032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f9e980096a0 tx=0x7f9e98006b40).stop 2023-10-12T22:36:29.803 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.801+0000 7f9eac6ca700 1 -- 172.21.15.195:0/3317868240 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea40f6510 msgr2=0x7f9ea406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:29.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.801+0000 7f9eac6ca700 1 --2- 172.21.15.195:0/3317868240 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea40f6510 0x7f9ea406b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f9e9c013510 tx=0x7f9e9c00b040).stop 2023-10-12T22:36:29.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.801+0000 7f9eac6ca700 1 -- 172.21.15.195:0/3317868240 shutdown_connections 2023-10-12T22:36:29.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.801+0000 7f9eac6ca700 1 --2- 172.21.15.195:0/3317868240 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9e8c02ff60 0x7f9e8c032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:29.804 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.801+0000 7f9eac6ca700 1 --2- 172.21.15.195:0/3317868240 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea40f6510 0x7f9ea406b9a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:29.805 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.801+0000 7f9eac6ca700 1 -- 172.21.15.195:0/3317868240 >> 172.21.15.195:0/3317868240 conn(0x7f9ea40f1c70 msgr2=0x7f9ea40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:29.805 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.801+0000 7f9eac6ca700 1 -- 172.21.15.195:0/3317868240 shutdown_connections 2023-10-12T22:36:29.805 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:29.801+0000 7f9eac6ca700 1 -- 172.21.15.195:0/3317868240 wait complete. 2023-10-12T22:36:29.805 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:30.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:30 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:28.844608+0000 mgr.smithi131.efktma (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:30.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:30 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:29.800473+0000 mon.smithi131 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.195:0/3317868240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:31.226 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:31.226 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:32.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.317+0000 7f868f59e700 1 -- 172.21.15.195:0/2578017379 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8690060180 msgr2=0x7f86900f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:32.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.317+0000 7f868f59e700 1 --2- 172.21.15.195:0/2578017379 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8690060180 0x7f86900f8f80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f8680009fb0 tx=0x7f8680009b30).stop 2023-10-12T22:36:32.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.317+0000 7f868f59e700 1 -- 172.21.15.195:0/2578017379 shutdown_connections 2023-10-12T22:36:32.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.317+0000 7f868f59e700 1 --2- 172.21.15.195:0/2578017379 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8690060180 0x7f86900f8f80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:32.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.317+0000 7f868f59e700 1 -- 172.21.15.195:0/2578017379 >> 172.21.15.195:0/2578017379 conn(0x7f86900f09a0 msgr2=0x7f86900f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:32.321 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.317+0000 7f868f59e700 1 -- 172.21.15.195:0/2578017379 shutdown_connections 2023-10-12T22:36:32.321 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.317+0000 7f868f59e700 1 -- 172.21.15.195:0/2578017379 wait complete. 2023-10-12T22:36:32.321 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.317+0000 7f868f59e700 1 Processor -- start 2023-10-12T22:36:32.321 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.317+0000 7f868f59e700 1 -- start start 2023-10-12T22:36:32.321 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.318+0000 7f868f59e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8690060180 0x7f86901064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:32.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.318+0000 7f868f59e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86901069e0 con 0x7f8690060180 2023-10-12T22:36:32.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.318+0000 7f868e59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8690060180 0x7f86901064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:32.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.318+0000 7f868e59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8690060180 0x7f86901064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38598/0 (socket says 172.21.15.195:38598) 2023-10-12T22:36:32.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.318+0000 7f868e59c700 1 -- 172.21.15.195:0/1562588272 learned_addr learned my addr 172.21.15.195:0/1562588272 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:32.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.318+0000 7f868e59c700 1 -- 172.21.15.195:0/1562588272 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8680005040 con 0x7f8690060180 2023-10-12T22:36:32.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.318+0000 7f868e59c700 1 --2- 172.21.15.195:0/1562588272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8690060180 0x7f86901064f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f8680013510 tx=0x7f868000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:32.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.318+0000 7f867effd700 1 -- 172.21.15.195:0/1562588272 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8680009320 con 0x7f8690060180 2023-10-12T22:36:32.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.318+0000 7f868f59e700 1 -- 172.21.15.195:0/1562588272 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8690106be0 con 0x7f8690060180 2023-10-12T22:36:32.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.318+0000 7f868f59e700 1 -- 172.21.15.195:0/1562588272 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8690107070 con 0x7f8690060180 2023-10-12T22:36:32.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.318+0000 7f867effd700 1 -- 172.21.15.195:0/1562588272 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8680013f00 con 0x7f8690060180 2023-10-12T22:36:32.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.318+0000 7f867effd700 1 -- 172.21.15.195:0/1562588272 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8680014d90 con 0x7f8690060180 2023-10-12T22:36:32.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.319+0000 7f867effd700 1 -- 172.21.15.195:0/1562588272 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86800263a0 con 0x7f8690060180 2023-10-12T22:36:32.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.319+0000 7f867effd700 1 --2- 172.21.15.195:0/1562588272 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f867802ffb0 0x7f8678032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:32.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.319+0000 7f867effd700 1 -- 172.21.15.195:0/1562588272 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8680041010 con 0x7f8690060180 2023-10-12T22:36:32.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.320+0000 7f868f59e700 1 -- 172.21.15.195:0/1562588272 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8674004fa0 con 0x7f8690060180 2023-10-12T22:36:32.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.320+0000 7f868dd9b700 1 --2- 172.21.15.195:0/1562588272 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f867802ffb0 0x7f8678032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:32.326 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.320+0000 7f868dd9b700 1 --2- 172.21.15.195:0/1562588272 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f867802ffb0 0x7f8678032470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f86840096a0 tx=0x7f8684006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:32.326 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.324+0000 7f867effd700 1 -- 172.21.15.195:0/1562588272 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8680012920 con 0x7f8690060180 2023-10-12T22:36:32.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:30.844941+0000 mgr.smithi131.efktma (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:32.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.531+0000 7f868f59e700 1 -- 172.21.15.195:0/1562588272 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8674005670 con 0x7f8690060180 2023-10-12T22:36:32.533 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.531+0000 7f867effd700 1 -- 172.21.15.195:0/1562588272 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f868002b020 con 0x7f8690060180 2023-10-12T22:36:32.533 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:32.533 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:32.534 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.533+0000 7f868f59e700 1 -- 172.21.15.195:0/1562588272 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f867802ffb0 msgr2=0x7f8678032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:32.534 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.533+0000 7f868f59e700 1 --2- 172.21.15.195:0/1562588272 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f867802ffb0 0x7f8678032470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f86840096a0 tx=0x7f8684006b40).stop 2023-10-12T22:36:32.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.533+0000 7f868f59e700 1 -- 172.21.15.195:0/1562588272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8690060180 msgr2=0x7f86901064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:32.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.533+0000 7f868f59e700 1 --2- 172.21.15.195:0/1562588272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8690060180 0x7f86901064f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f8680013510 tx=0x7f868000b040).stop 2023-10-12T22:36:32.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.533+0000 7f868f59e700 1 -- 172.21.15.195:0/1562588272 shutdown_connections 2023-10-12T22:36:32.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.533+0000 7f868f59e700 1 --2- 172.21.15.195:0/1562588272 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f867802ffb0 0x7f8678032470 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:32.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.533+0000 7f868f59e700 1 --2- 172.21.15.195:0/1562588272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8690060180 0x7f86901064f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:32.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.533+0000 7f868f59e700 1 -- 172.21.15.195:0/1562588272 >> 172.21.15.195:0/1562588272 conn(0x7f86900f09a0 msgr2=0x7f86900f1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:32.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.533+0000 7f868f59e700 1 -- 172.21.15.195:0/1562588272 shutdown_connections 2023-10-12T22:36:32.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:32.533+0000 7f868f59e700 1 -- 172.21.15.195:0/1562588272 wait complete. 2023-10-12T22:36:32.536 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:33.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:33 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:32.532938+0000 mon.smithi131 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.195:0/1562588272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:33.950 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:33.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:34.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:32.845316+0000 mgr.smithi131.efktma (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:35.061 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.058+0000 7fe787c0b700 1 -- 172.21.15.195:0/4159228032 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7800f6510 msgr2=0x7fe7800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:35.061 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.058+0000 7fe787c0b700 1 --2- 172.21.15.195:0/4159228032 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7800f6510 0x7fe7800f6930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fe770009960 tx=0x7fe770004d10).stop 2023-10-12T22:36:35.061 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.058+0000 7fe787c0b700 1 -- 172.21.15.195:0/4159228032 shutdown_connections 2023-10-12T22:36:35.062 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.058+0000 7fe787c0b700 1 --2- 172.21.15.195:0/4159228032 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7800f6510 0x7fe7800f6930 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:35.062 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.058+0000 7fe787c0b700 1 -- 172.21.15.195:0/4159228032 >> 172.21.15.195:0/4159228032 conn(0x7fe7800f1c70 msgr2=0x7fe7800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:35.062 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.058+0000 7fe787c0b700 1 -- 172.21.15.195:0/4159228032 shutdown_connections 2023-10-12T22:36:35.062 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.058+0000 7fe787c0b700 1 -- 172.21.15.195:0/4159228032 wait complete. 2023-10-12T22:36:35.062 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.059+0000 7fe787c0b700 1 Processor -- start 2023-10-12T22:36:35.063 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.059+0000 7fe787c0b700 1 -- start start 2023-10-12T22:36:35.063 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.059+0000 7fe787c0b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7800f6510 0x7fe780106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:35.063 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.059+0000 7fe787c0b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe780106c20 con 0x7fe7800f6510 2023-10-12T22:36:35.063 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.059+0000 7fe7859a7700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7800f6510 0x7fe780106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:35.063 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.059+0000 7fe7859a7700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7800f6510 0x7fe780106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38626/0 (socket says 172.21.15.195:38626) 2023-10-12T22:36:35.064 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.059+0000 7fe7859a7700 1 -- 172.21.15.195:0/1669865357 learned_addr learned my addr 172.21.15.195:0/1669865357 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:35.064 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.060+0000 7fe7859a7700 1 -- 172.21.15.195:0/1669865357 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe770009640 con 0x7fe7800f6510 2023-10-12T22:36:35.064 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.060+0000 7fe7859a7700 1 --2- 172.21.15.195:0/1669865357 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7800f6510 0x7fe780106730 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fe770004b70 tx=0x7fe770004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:35.064 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.060+0000 7fe77e7fc700 1 -- 172.21.15.195:0/1669865357 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe770018070 con 0x7fe7800f6510 2023-10-12T22:36:35.064 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.060+0000 7fe787c0b700 1 -- 172.21.15.195:0/1669865357 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe780106e20 con 0x7fe7800f6510 2023-10-12T22:36:35.065 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.060+0000 7fe77e7fc700 1 -- 172.21.15.195:0/1669865357 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe77000b870 con 0x7fe7800f6510 2023-10-12T22:36:35.065 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.060+0000 7fe77e7fc700 1 -- 172.21.15.195:0/1669865357 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe77001c560 con 0x7fe7800f6510 2023-10-12T22:36:35.065 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.060+0000 7fe787c0b700 1 -- 172.21.15.195:0/1669865357 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7801072b0 con 0x7fe7800f6510 2023-10-12T22:36:35.066 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.061+0000 7fe77e7fc700 1 -- 172.21.15.195:0/1669865357 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe770013630 con 0x7fe7800f6510 2023-10-12T22:36:35.066 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.061+0000 7fe77e7fc700 1 --2- 172.21.15.195:0/1669865357 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe76802ff60 0x7fe768032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:35.066 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.061+0000 7fe787c0b700 1 -- 172.21.15.195:0/1669865357 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe780045ad0 con 0x7fe7800f6510 2023-10-12T22:36:35.067 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.061+0000 7fe77e7fc700 1 -- 172.21.15.195:0/1669865357 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe770041330 con 0x7fe7800f6510 2023-10-12T22:36:35.067 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.061+0000 7fe7851a6700 1 --2- 172.21.15.195:0/1669865357 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe76802ff60 0x7fe768032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:35.067 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.061+0000 7fe7851a6700 1 --2- 172.21.15.195:0/1669865357 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe76802ff60 0x7fe768032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fe7740096a0 tx=0x7fe774006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:35.067 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.066+0000 7fe77e7fc700 1 -- 172.21.15.195:0/1669865357 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe770004810 con 0x7fe7800f6510 2023-10-12T22:36:35.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.274+0000 7fe787c0b700 1 -- 172.21.15.195:0/1669865357 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7800fc050 con 0x7fe7800f6510 2023-10-12T22:36:35.277 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.275+0000 7fe77e7fc700 1 -- 172.21.15.195:0/1669865357 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe770012380 con 0x7fe7800f6510 2023-10-12T22:36:35.277 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:35.278 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:35.279 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.277+0000 7fe787c0b700 1 -- 172.21.15.195:0/1669865357 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe76802ff60 msgr2=0x7fe768032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:35.279 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.277+0000 7fe787c0b700 1 --2- 172.21.15.195:0/1669865357 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe76802ff60 0x7fe768032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fe7740096a0 tx=0x7fe774006b40).stop 2023-10-12T22:36:35.280 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.277+0000 7fe787c0b700 1 -- 172.21.15.195:0/1669865357 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7800f6510 msgr2=0x7fe780106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:35.280 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.277+0000 7fe787c0b700 1 --2- 172.21.15.195:0/1669865357 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7800f6510 0x7fe780106730 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fe770004b70 tx=0x7fe770004d10).stop 2023-10-12T22:36:35.280 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.277+0000 7fe787c0b700 1 -- 172.21.15.195:0/1669865357 shutdown_connections 2023-10-12T22:36:35.281 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.277+0000 7fe787c0b700 1 --2- 172.21.15.195:0/1669865357 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe76802ff60 0x7fe768032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:35.281 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.277+0000 7fe787c0b700 1 --2- 172.21.15.195:0/1669865357 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7800f6510 0x7fe780106730 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:35.282 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.277+0000 7fe787c0b700 1 -- 172.21.15.195:0/1669865357 >> 172.21.15.195:0/1669865357 conn(0x7fe7800f1c70 msgr2=0x7fe7800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:35.282 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.277+0000 7fe787c0b700 1 -- 172.21.15.195:0/1669865357 shutdown_connections 2023-10-12T22:36:35.282 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:35.277+0000 7fe787c0b700 1 -- 172.21.15.195:0/1669865357 wait complete. 2023-10-12T22:36:35.283 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:36.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:36 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:34.845585+0000 mgr.smithi131.efktma (mgr.14162) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:36.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:36 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:35.276362+0000 mon.smithi131 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.195:0/1669865357' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:36.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:36.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:37.869 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.867+0000 7f32e27e5700 1 -- 172.21.15.195:0/1008162783 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32dc0f6510 msgr2=0x7f32dc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:37.869 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.867+0000 7f32e27e5700 1 --2- 172.21.15.195:0/1008162783 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32dc0f6510 0x7f32dc0f6930 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f32cc009fb0 tx=0x7f32cc009b30).stop 2023-10-12T22:36:37.869 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.867+0000 7f32e27e5700 1 -- 172.21.15.195:0/1008162783 shutdown_connections 2023-10-12T22:36:37.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.867+0000 7f32e27e5700 1 --2- 172.21.15.195:0/1008162783 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32dc0f6510 0x7f32dc0f6930 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:37.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.867+0000 7f32e27e5700 1 -- 172.21.15.195:0/1008162783 >> 172.21.15.195:0/1008162783 conn(0x7f32dc0f1c70 msgr2=0x7f32dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:37.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.867+0000 7f32e27e5700 1 -- 172.21.15.195:0/1008162783 shutdown_connections 2023-10-12T22:36:37.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.867+0000 7f32e27e5700 1 -- 172.21.15.195:0/1008162783 wait complete. 2023-10-12T22:36:37.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.867+0000 7f32e27e5700 1 Processor -- start 2023-10-12T22:36:37.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.868+0000 7f32e27e5700 1 -- start start 2023-10-12T22:36:37.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.868+0000 7f32e27e5700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32dc0f6510 0x7f32dc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:37.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.868+0000 7f32e27e5700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32dc106c20 con 0x7f32dc0f6510 2023-10-12T22:36:37.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.868+0000 7f32dbfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32dc0f6510 0x7f32dc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:37.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.868+0000 7f32dbfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32dc0f6510 0x7f32dc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38632/0 (socket says 172.21.15.195:38632) 2023-10-12T22:36:37.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.868+0000 7f32dbfff700 1 -- 172.21.15.195:0/4221457018 learned_addr learned my addr 172.21.15.195:0/4221457018 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:37.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.869+0000 7f32dbfff700 1 -- 172.21.15.195:0/4221457018 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc005040 con 0x7f32dc0f6510 2023-10-12T22:36:37.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.869+0000 7f32dbfff700 1 --2- 172.21.15.195:0/4221457018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32dc0f6510 0x7f32dc106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f32cc013510 tx=0x7f32cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:37.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.869+0000 7f32d8ff9700 1 -- 172.21.15.195:0/4221457018 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32cc009320 con 0x7f32dc0f6510 2023-10-12T22:36:37.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.869+0000 7f32e27e5700 1 -- 172.21.15.195:0/4221457018 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32dc106e20 con 0x7f32dc0f6510 2023-10-12T22:36:37.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.869+0000 7f32d8ff9700 1 -- 172.21.15.195:0/4221457018 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32cc013f00 con 0x7f32dc0f6510 2023-10-12T22:36:37.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.869+0000 7f32d8ff9700 1 -- 172.21.15.195:0/4221457018 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32cc014d90 con 0x7f32dc0f6510 2023-10-12T22:36:37.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.869+0000 7f32e27e5700 1 -- 172.21.15.195:0/4221457018 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32dc1072b0 con 0x7f32dc0f6510 2023-10-12T22:36:37.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.870+0000 7f32d8ff9700 1 -- 172.21.15.195:0/4221457018 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32cc014420 con 0x7f32dc0f6510 2023-10-12T22:36:37.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.870+0000 7f32d8ff9700 1 --2- 172.21.15.195:0/4221457018 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f32c802ff60 0x7f32c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:37.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.870+0000 7f32d8ff9700 1 -- 172.21.15.195:0/4221457018 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32cc041000 con 0x7f32dc0f6510 2023-10-12T22:36:37.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.870+0000 7f32e27e5700 1 -- 172.21.15.195:0/4221457018 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32dc045ad0 con 0x7f32dc0f6510 2023-10-12T22:36:37.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.870+0000 7f32db7fe700 1 --2- 172.21.15.195:0/4221457018 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f32c802ff60 0x7f32c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:37.877 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.870+0000 7f32db7fe700 1 --2- 172.21.15.195:0/4221457018 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f32c802ff60 0x7f32c8032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f32c40096a0 tx=0x7f32c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:37.878 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:37.875+0000 7f32d8ff9700 1 -- 172.21.15.195:0/4221457018 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32cc0095f0 con 0x7f32dc0f6510 2023-10-12T22:36:38.083 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:38.082+0000 7f32e27e5700 1 -- 172.21.15.195:0/4221457018 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32dc0fbf60 con 0x7f32dc0f6510 2023-10-12T22:36:38.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:38.083+0000 7f32d8ff9700 1 -- 172.21.15.195:0/4221457018 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f32cc01db70 con 0x7f32dc0f6510 2023-10-12T22:36:38.084 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:38.084 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:38.085 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:38.084+0000 7f32e27e5700 1 -- 172.21.15.195:0/4221457018 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f32c802ff60 msgr2=0x7f32c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:38.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:38.084+0000 7f32e27e5700 1 --2- 172.21.15.195:0/4221457018 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f32c802ff60 0x7f32c8032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f32c40096a0 tx=0x7f32c4006b40).stop 2023-10-12T22:36:38.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:38.084+0000 7f32e27e5700 1 -- 172.21.15.195:0/4221457018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32dc0f6510 msgr2=0x7f32dc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:38.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:38.084+0000 7f32e27e5700 1 --2- 172.21.15.195:0/4221457018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32dc0f6510 0x7f32dc106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f32cc013510 tx=0x7f32cc00b040).stop 2023-10-12T22:36:38.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:38.084+0000 7f32e27e5700 1 -- 172.21.15.195:0/4221457018 shutdown_connections 2023-10-12T22:36:38.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:38.084+0000 7f32e27e5700 1 --2- 172.21.15.195:0/4221457018 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f32c802ff60 0x7f32c8032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:38.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:38.084+0000 7f32e27e5700 1 --2- 172.21.15.195:0/4221457018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32dc0f6510 0x7f32dc106730 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:38.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:38.084+0000 7f32e27e5700 1 -- 172.21.15.195:0/4221457018 >> 172.21.15.195:0/4221457018 conn(0x7f32dc0f1c70 msgr2=0x7f32dc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:38.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:38.084+0000 7f32e27e5700 1 -- 172.21.15.195:0/4221457018 shutdown_connections 2023-10-12T22:36:38.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:38.084+0000 7f32e27e5700 1 -- 172.21.15.195:0/4221457018 wait complete. 2023-10-12T22:36:38.088 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:38.492 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:38 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:36.845821+0000 mgr.smithi131.efktma (mgr.14162) 158 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:39.427 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:39.428 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:39.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:39 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:38.084197+0000 mon.smithi131 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.195:0/4221457018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:40.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:40 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:38.846186+0000 mgr.smithi131.efktma (mgr.14162) 159 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:40.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.637+0000 7f40c2995700 1 -- 172.21.15.195:0/1523017055 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40bc0f5410 msgr2=0x7f40bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:40.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.637+0000 7f40c2995700 1 --2- 172.21.15.195:0/1523017055 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40bc0f5410 0x7f40bc0f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f40b0009fb0 tx=0x7f40b0009b30).stop 2023-10-12T22:36:40.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.637+0000 7f40c2995700 1 -- 172.21.15.195:0/1523017055 shutdown_connections 2023-10-12T22:36:40.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.637+0000 7f40c2995700 1 --2- 172.21.15.195:0/1523017055 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40bc0f5410 0x7f40bc0f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:40.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.637+0000 7f40c2995700 1 -- 172.21.15.195:0/1523017055 >> 172.21.15.195:0/1523017055 conn(0x7f40bc0f0b70 msgr2=0x7f40bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:40.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.637+0000 7f40c2995700 1 -- 172.21.15.195:0/1523017055 shutdown_connections 2023-10-12T22:36:40.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.637+0000 7f40c2995700 1 -- 172.21.15.195:0/1523017055 wait complete. 2023-10-12T22:36:40.644 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.638+0000 7f40c2995700 1 Processor -- start 2023-10-12T22:36:40.645 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.638+0000 7f40c2995700 1 -- start start 2023-10-12T22:36:40.645 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.638+0000 7f40c2995700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40bc0f5410 0x7f40bc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:40.645 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.638+0000 7f40c2995700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc060240 con 0x7f40bc0f5410 2023-10-12T22:36:40.645 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.638+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40bc0f5410 0x7f40bc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:40.645 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.638+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40bc0f5410 0x7f40bc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51690/0 (socket says 172.21.15.195:51690) 2023-10-12T22:36:40.646 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.638+0000 7f40bbfff700 1 -- 172.21.15.195:0/612010668 learned_addr learned my addr 172.21.15.195:0/612010668 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:40.646 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.639+0000 7f40bbfff700 1 -- 172.21.15.195:0/612010668 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b0005040 con 0x7f40bc0f5410 2023-10-12T22:36:40.646 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.639+0000 7f40bbfff700 1 --2- 172.21.15.195:0/612010668 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40bc0f5410 0x7f40bc109430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f40b0013510 tx=0x7f40b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:40.646 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.639+0000 7f40b8ff9700 1 -- 172.21.15.195:0/612010668 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40b0009320 con 0x7f40bc0f5410 2023-10-12T22:36:40.646 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.639+0000 7f40c2995700 1 -- 172.21.15.195:0/612010668 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc060440 con 0x7f40bc0f5410 2023-10-12T22:36:40.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.639+0000 7f40b8ff9700 1 -- 172.21.15.195:0/612010668 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40b0013f00 con 0x7f40bc0f5410 2023-10-12T22:36:40.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.639+0000 7f40b8ff9700 1 -- 172.21.15.195:0/612010668 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40b0014d90 con 0x7f40bc0f5410 2023-10-12T22:36:40.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.639+0000 7f40c2995700 1 -- 172.21.15.195:0/612010668 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc109ae0 con 0x7f40bc0f5410 2023-10-12T22:36:40.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.640+0000 7f40b8ff9700 1 -- 172.21.15.195:0/612010668 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40b0014420 con 0x7f40bc0f5410 2023-10-12T22:36:40.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.640+0000 7f40c2995700 1 -- 172.21.15.195:0/612010668 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40bc045ad0 con 0x7f40bc0f5410 2023-10-12T22:36:40.647 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.640+0000 7f40b8ff9700 1 --2- 172.21.15.195:0/612010668 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f40a402ff60 0x7f40a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:40.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.640+0000 7f40b8ff9700 1 -- 172.21.15.195:0/612010668 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40b00410c0 con 0x7f40bc0f5410 2023-10-12T22:36:40.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.640+0000 7f40bb7fe700 1 --2- 172.21.15.195:0/612010668 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f40a402ff60 0x7f40a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:40.648 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.641+0000 7f40bb7fe700 1 --2- 172.21.15.195:0/612010668 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f40a402ff60 0x7f40a4032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f40ac0096a0 tx=0x7f40ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:40.649 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.645+0000 7f40b8ff9700 1 -- 172.21.15.195:0/612010668 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40b00095f0 con 0x7f40bc0f5410 2023-10-12T22:36:40.851 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.850+0000 7f40c2995700 1 -- 172.21.15.195:0/612010668 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40bc0fae60 con 0x7f40bc0f5410 2023-10-12T22:36:40.852 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.850+0000 7f40b8ff9700 1 -- 172.21.15.195:0/612010668 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f40b001db70 con 0x7f40bc0f5410 2023-10-12T22:36:40.852 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:40.852 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:40.853 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.852+0000 7f40c2995700 1 -- 172.21.15.195:0/612010668 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f40a402ff60 msgr2=0x7f40a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:40.853 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.852+0000 7f40c2995700 1 --2- 172.21.15.195:0/612010668 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f40a402ff60 0x7f40a4032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f40ac0096a0 tx=0x7f40ac006b40).stop 2023-10-12T22:36:40.854 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.852+0000 7f40c2995700 1 -- 172.21.15.195:0/612010668 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40bc0f5410 msgr2=0x7f40bc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:40.854 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.852+0000 7f40c2995700 1 --2- 172.21.15.195:0/612010668 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40bc0f5410 0x7f40bc109430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f40b0013510 tx=0x7f40b000b040).stop 2023-10-12T22:36:40.854 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.852+0000 7f40c2995700 1 -- 172.21.15.195:0/612010668 shutdown_connections 2023-10-12T22:36:40.854 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.852+0000 7f40c2995700 1 --2- 172.21.15.195:0/612010668 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f40a402ff60 0x7f40a4032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:40.854 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.852+0000 7f40c2995700 1 --2- 172.21.15.195:0/612010668 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40bc0f5410 0x7f40bc109430 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:40.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.852+0000 7f40c2995700 1 -- 172.21.15.195:0/612010668 >> 172.21.15.195:0/612010668 conn(0x7f40bc0f0b70 msgr2=0x7f40bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:40.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.852+0000 7f40c2995700 1 -- 172.21.15.195:0/612010668 shutdown_connections 2023-10-12T22:36:40.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:40.852+0000 7f40c2995700 1 -- 172.21.15.195:0/612010668 wait complete. 2023-10-12T22:36:40.855 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:41.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:40.851852+0000 mon.smithi131 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.195:0/612010668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:42.248 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:42.249 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:42.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:42 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:40.846384+0000 mgr.smithi131.efktma (mgr.14162) 160 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:43.386 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.385+0000 7f16ed127700 1 -- 172.21.15.195:0/1869030327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16e80f6510 msgr2=0x7f16e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:43.387 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.385+0000 7f16ed127700 1 --2- 172.21.15.195:0/1869030327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16e80f6510 0x7f16e80f6930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f16dc009fb0 tx=0x7f16dc009b30).stop 2023-10-12T22:36:43.387 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.385+0000 7f16ed127700 1 -- 172.21.15.195:0/1869030327 shutdown_connections 2023-10-12T22:36:43.387 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.385+0000 7f16ed127700 1 --2- 172.21.15.195:0/1869030327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16e80f6510 0x7f16e80f6930 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:43.387 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.385+0000 7f16ed127700 1 -- 172.21.15.195:0/1869030327 >> 172.21.15.195:0/1869030327 conn(0x7f16e80f1c70 msgr2=0x7f16e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:43.388 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.385+0000 7f16ed127700 1 -- 172.21.15.195:0/1869030327 shutdown_connections 2023-10-12T22:36:43.388 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.385+0000 7f16ed127700 1 -- 172.21.15.195:0/1869030327 wait complete. 2023-10-12T22:36:43.390 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.386+0000 7f16ed127700 1 Processor -- start 2023-10-12T22:36:43.390 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.386+0000 7f16ed127700 1 -- start start 2023-10-12T22:36:43.390 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.386+0000 7f16ed127700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:43.391 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.386+0000 7f16ed127700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e8106c20 con 0x7f16e80f6510 2023-10-12T22:36:43.391 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.386+0000 7f16e6d9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:43.391 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.386+0000 7f16e6d9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51708/0 (socket says 172.21.15.195:51708) 2023-10-12T22:36:43.391 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.386+0000 7f16e6d9d700 1 -- 172.21.15.195:0/2391147383 learned_addr learned my addr 172.21.15.195:0/2391147383 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:43.391 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.387+0000 7f16e6d9d700 1 -- 172.21.15.195:0/2391147383 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc005040 con 0x7f16e80f6510 2023-10-12T22:36:43.392 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.387+0000 7f16e6d9d700 1 --2- 172.21.15.195:0/2391147383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f16dc013510 tx=0x7f16dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:43.392 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.387+0000 7f16cf7fe700 1 -- 172.21.15.195:0/2391147383 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16dc009320 con 0x7f16e80f6510 2023-10-12T22:36:43.392 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.387+0000 7f16ed127700 1 -- 172.21.15.195:0/2391147383 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e8106e20 con 0x7f16e80f6510 2023-10-12T22:36:43.392 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.387+0000 7f16cf7fe700 1 -- 172.21.15.195:0/2391147383 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16dc013f00 con 0x7f16e80f6510 2023-10-12T22:36:43.392 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.387+0000 7f16cf7fe700 1 -- 172.21.15.195:0/2391147383 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16dc014d90 con 0x7f16e80f6510 2023-10-12T22:36:43.392 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.387+0000 7f16ed127700 1 -- 172.21.15.195:0/2391147383 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e81072b0 con 0x7f16e80f6510 2023-10-12T22:36:43.393 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.388+0000 7f16cf7fe700 1 -- 172.21.15.195:0/2391147383 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16dc0095f0 con 0x7f16e80f6510 2023-10-12T22:36:43.393 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.388+0000 7f16ed127700 1 -- 172.21.15.195:0/2391147383 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e8045ad0 con 0x7f16e80f6510 2023-10-12T22:36:43.393 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.388+0000 7f16cf7fe700 1 --2- 172.21.15.195:0/2391147383 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f16d002ff60 0x7f16d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:43.393 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.388+0000 7f16cf7fe700 1 -- 172.21.15.195:0/2391147383 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16dc0410a0 con 0x7f16e80f6510 2023-10-12T22:36:43.393 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.389+0000 7f16e659c700 1 --2- 172.21.15.195:0/2391147383 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f16d002ff60 0x7f16d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:43.394 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.389+0000 7f16e659c700 1 --2- 172.21.15.195:0/2391147383 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f16d002ff60 0x7f16d0032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f16d80096a0 tx=0x7f16d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:43.394 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.393+0000 7f16cf7fe700 1 -- 172.21.15.195:0/2391147383 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16dc014420 con 0x7f16e80f6510 2023-10-12T22:36:43.602 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.601+0000 7f16ed127700 1 -- 172.21.15.195:0/2391147383 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16e80fbf60 con 0x7f16e80f6510 2023-10-12T22:36:43.603 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.602+0000 7f16cf7fe700 1 -- 172.21.15.195:0/2391147383 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f16dc010230 con 0x7f16e80f6510 2023-10-12T22:36:43.603 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:43.604 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:43.605 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.604+0000 7f16ed127700 1 -- 172.21.15.195:0/2391147383 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f16d002ff60 msgr2=0x7f16d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:43.605 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.604+0000 7f16ed127700 1 --2- 172.21.15.195:0/2391147383 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f16d002ff60 0x7f16d0032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f16d80096a0 tx=0x7f16d8006b40).stop 2023-10-12T22:36:43.605 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.604+0000 7f16ed127700 1 -- 172.21.15.195:0/2391147383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16e80f6510 msgr2=0x7f16e8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:43.605 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.604+0000 7f16ed127700 1 --2- 172.21.15.195:0/2391147383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f16dc013510 tx=0x7f16dc00b040).stop 2023-10-12T22:36:43.606 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.604+0000 7f16ed127700 1 -- 172.21.15.195:0/2391147383 shutdown_connections 2023-10-12T22:36:43.606 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.604+0000 7f16ed127700 1 --2- 172.21.15.195:0/2391147383 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f16d002ff60 0x7f16d0032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:43.606 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.604+0000 7f16ed127700 1 --2- 172.21.15.195:0/2391147383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:43.606 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.604+0000 7f16ed127700 1 -- 172.21.15.195:0/2391147383 >> 172.21.15.195:0/2391147383 conn(0x7f16e80f1c70 msgr2=0x7f16e80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:43.606 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.604+0000 7f16ed127700 1 -- 172.21.15.195:0/2391147383 shutdown_connections 2023-10-12T22:36:43.607 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:43.604+0000 7f16ed127700 1 -- 172.21.15.195:0/2391147383 wait complete. 2023-10-12T22:36:43.607 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:44.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:44 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:42.846646+0000 mgr.smithi131.efktma (mgr.14162) 161 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:44.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:44 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:43.603420+0000 mon.smithi131 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.195:0/2391147383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:44.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:44.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:46.105 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.103+0000 7f032c9f8700 1 -- 172.21.15.195:0/2194127592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03280f62a0 msgr2=0x7f03280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:46.105 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.103+0000 7f032c9f8700 1 --2- 172.21.15.195:0/2194127592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03280f62a0 0x7f03280f66c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f0310009fb0 tx=0x7f0310009b30).stop 2023-10-12T22:36:46.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.103+0000 7f032c9f8700 1 -- 172.21.15.195:0/2194127592 shutdown_connections 2023-10-12T22:36:46.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.103+0000 7f032c9f8700 1 --2- 172.21.15.195:0/2194127592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03280f62a0 0x7f03280f66c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:46.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.103+0000 7f032c9f8700 1 -- 172.21.15.195:0/2194127592 >> 172.21.15.195:0/2194127592 conn(0x7f03280f1aa0 msgr2=0x7f03280f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:46.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.103+0000 7f032c9f8700 1 -- 172.21.15.195:0/2194127592 shutdown_connections 2023-10-12T22:36:46.107 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.103+0000 7f032c9f8700 1 -- 172.21.15.195:0/2194127592 wait complete. 2023-10-12T22:36:46.107 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.104+0000 7f032c9f8700 1 Processor -- start 2023-10-12T22:36:46.107 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.104+0000 7f032c9f8700 1 -- start start 2023-10-12T22:36:46.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.104+0000 7f032c9f8700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03280f62a0 0x7f03281064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:46.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.104+0000 7f032c9f8700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03281069a0 con 0x7f03280f62a0 2023-10-12T22:36:46.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.104+0000 7f03277fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03280f62a0 0x7f03281064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:46.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.104+0000 7f03277fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03280f62a0 0x7f03281064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51724/0 (socket says 172.21.15.195:51724) 2023-10-12T22:36:46.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.104+0000 7f03277fe700 1 -- 172.21.15.195:0/1098862874 learned_addr learned my addr 172.21.15.195:0/1098862874 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:46.110 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.105+0000 7f03277fe700 1 -- 172.21.15.195:0/1098862874 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0310005040 con 0x7f03280f62a0 2023-10-12T22:36:46.110 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.105+0000 7f03277fe700 1 --2- 172.21.15.195:0/1098862874 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03280f62a0 0x7f03281064b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f0310009830 tx=0x7f031000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:46.110 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.105+0000 7f031ffff700 1 -- 172.21.15.195:0/1098862874 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0310009320 con 0x7f03280f62a0 2023-10-12T22:36:46.110 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.105+0000 7f031ffff700 1 -- 172.21.15.195:0/1098862874 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03100136c0 con 0x7f03280f62a0 2023-10-12T22:36:46.111 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.105+0000 7f031ffff700 1 -- 172.21.15.195:0/1098862874 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f031001ee90 con 0x7f03280f62a0 2023-10-12T22:36:46.111 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.105+0000 7f032c9f8700 1 -- 172.21.15.195:0/1098862874 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0328106ba0 con 0x7f03280f62a0 2023-10-12T22:36:46.111 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.105+0000 7f032c9f8700 1 -- 172.21.15.195:0/1098862874 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0328107030 con 0x7f03280f62a0 2023-10-12T22:36:46.111 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.106+0000 7f031ffff700 1 -- 172.21.15.195:0/1098862874 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f031001e420 con 0x7f03280f62a0 2023-10-12T22:36:46.111 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.106+0000 7f031ffff700 1 --2- 172.21.15.195:0/1098862874 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f031402ff10 0x7f03140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:46.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.106+0000 7f032c9f8700 1 -- 172.21.15.195:0/1098862874 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0328045ad0 con 0x7f03280f62a0 2023-10-12T22:36:46.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.106+0000 7f031ffff700 1 -- 172.21.15.195:0/1098862874 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0310040f70 con 0x7f03280f62a0 2023-10-12T22:36:46.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.106+0000 7f0326ffd700 1 --2- 172.21.15.195:0/1098862874 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f031402ff10 0x7f03140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:46.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.107+0000 7f0326ffd700 1 --2- 172.21.15.195:0/1098862874 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f031402ff10 0x7f03140323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f03180096a0 tx=0x7f0318006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:46.113 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.111+0000 7f031ffff700 1 -- 172.21.15.195:0/1098862874 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03100095f0 con 0x7f03280f62a0 2023-10-12T22:36:46.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.317+0000 7f032c9f8700 1 -- 172.21.15.195:0/1098862874 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0328107ee0 con 0x7f03280f62a0 2023-10-12T22:36:46.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.318+0000 7f031ffff700 1 -- 172.21.15.195:0/1098862874 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f031000fe60 con 0x7f03280f62a0 2023-10-12T22:36:46.320 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:46.320 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:46.321 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.319+0000 7f032c9f8700 1 -- 172.21.15.195:0/1098862874 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f031402ff10 msgr2=0x7f03140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:46.321 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.319+0000 7f032c9f8700 1 --2- 172.21.15.195:0/1098862874 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f031402ff10 0x7f03140323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f03180096a0 tx=0x7f0318006b40).stop 2023-10-12T22:36:46.321 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.319+0000 7f032c9f8700 1 -- 172.21.15.195:0/1098862874 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03280f62a0 msgr2=0x7f03281064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:46.321 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.319+0000 7f032c9f8700 1 --2- 172.21.15.195:0/1098862874 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03280f62a0 0x7f03281064b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f0310009830 tx=0x7f031000b040).stop 2023-10-12T22:36:46.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.319+0000 7f032c9f8700 1 -- 172.21.15.195:0/1098862874 shutdown_connections 2023-10-12T22:36:46.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.319+0000 7f032c9f8700 1 --2- 172.21.15.195:0/1098862874 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f031402ff10 0x7f03140323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:46.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.319+0000 7f032c9f8700 1 --2- 172.21.15.195:0/1098862874 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03280f62a0 0x7f03281064b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:46.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.319+0000 7f032c9f8700 1 -- 172.21.15.195:0/1098862874 >> 172.21.15.195:0/1098862874 conn(0x7f03280f1aa0 msgr2=0x7f03280f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:46.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.320+0000 7f032c9f8700 1 -- 172.21.15.195:0/1098862874 shutdown_connections 2023-10-12T22:36:46.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:46.320+0000 7f032c9f8700 1 -- 172.21.15.195:0/1098862874 wait complete. 2023-10-12T22:36:46.323 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:46.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:46 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:44.846991+0000 mgr.smithi131.efktma (mgr.14162) 162 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:47.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:47 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:46.319317+0000 mon.smithi131 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.195:0/1098862874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:47.702 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:47.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:48.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:48 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:46.847319+0000 mgr.smithi131.efktma (mgr.14162) 163 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:48.846 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.844+0000 7fb1f45a6700 1 -- 172.21.15.195:0/2923112933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1ec0f6510 msgr2=0x7fb1ec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:48.846 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.844+0000 7fb1f45a6700 1 --2- 172.21.15.195:0/2923112933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1ec0f6510 0x7fb1ec0f6930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fb1e8009fb0 tx=0x7fb1e8009b30).stop 2023-10-12T22:36:48.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.844+0000 7fb1f45a6700 1 -- 172.21.15.195:0/2923112933 shutdown_connections 2023-10-12T22:36:48.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.844+0000 7fb1f45a6700 1 --2- 172.21.15.195:0/2923112933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1ec0f6510 0x7fb1ec0f6930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:48.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.844+0000 7fb1f45a6700 1 -- 172.21.15.195:0/2923112933 >> 172.21.15.195:0/2923112933 conn(0x7fb1ec0f1c70 msgr2=0x7fb1ec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:48.848 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.844+0000 7fb1f45a6700 1 -- 172.21.15.195:0/2923112933 shutdown_connections 2023-10-12T22:36:48.848 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.844+0000 7fb1f45a6700 1 -- 172.21.15.195:0/2923112933 wait complete. 2023-10-12T22:36:48.851 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.845+0000 7fb1f45a6700 1 Processor -- start 2023-10-12T22:36:48.852 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.845+0000 7fb1f45a6700 1 -- start start 2023-10-12T22:36:48.852 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.845+0000 7fb1f45a6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1ec0f6510 0x7fb1ec106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:48.853 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.845+0000 7fb1f45a6700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec106c20 con 0x7fb1ec0f6510 2023-10-12T22:36:48.853 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.845+0000 7fb1f2342700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1ec0f6510 0x7fb1ec106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:48.854 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.845+0000 7fb1f2342700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1ec0f6510 0x7fb1ec106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:50998/0 (socket says 172.21.15.195:50998) 2023-10-12T22:36:48.854 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.845+0000 7fb1f2342700 1 -- 172.21.15.195:0/4237971071 learned_addr learned my addr 172.21.15.195:0/4237971071 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:48.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.846+0000 7fb1f2342700 1 -- 172.21.15.195:0/4237971071 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e8005040 con 0x7fb1ec0f6510 2023-10-12T22:36:48.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.846+0000 7fb1f2342700 1 --2- 172.21.15.195:0/4237971071 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1ec0f6510 0x7fb1ec106730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb1e8014000 tx=0x7fb1e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:48.855 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.846+0000 7fb1e2ffd700 1 -- 172.21.15.195:0/4237971071 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e80095f0 con 0x7fb1ec0f6510 2023-10-12T22:36:48.856 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.846+0000 7fb1f45a6700 1 -- 172.21.15.195:0/4237971071 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ec106e20 con 0x7fb1ec0f6510 2023-10-12T22:36:48.856 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.846+0000 7fb1e2ffd700 1 -- 172.21.15.195:0/4237971071 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1e8007e70 con 0x7fb1ec0f6510 2023-10-12T22:36:48.856 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.846+0000 7fb1e2ffd700 1 -- 172.21.15.195:0/4237971071 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e8014ea0 con 0x7fb1ec0f6510 2023-10-12T22:36:48.857 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.846+0000 7fb1f45a6700 1 -- 172.21.15.195:0/4237971071 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ec1072b0 con 0x7fb1ec0f6510 2023-10-12T22:36:48.857 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.847+0000 7fb1f45a6700 1 -- 172.21.15.195:0/4237971071 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ec045ad0 con 0x7fb1ec0f6510 2023-10-12T22:36:48.857 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.847+0000 7fb1e2ffd700 1 -- 172.21.15.195:0/4237971071 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1e8025070 con 0x7fb1ec0f6510 2023-10-12T22:36:48.858 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.847+0000 7fb1e2ffd700 1 --2- 172.21.15.195:0/4237971071 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb1d802ff60 0x7fb1d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:48.858 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.847+0000 7fb1e2ffd700 1 -- 172.21.15.195:0/4237971071 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1e8040970 con 0x7fb1ec0f6510 2023-10-12T22:36:48.858 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.847+0000 7fb1f1b41700 1 --2- 172.21.15.195:0/4237971071 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb1d802ff60 0x7fb1d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:48.859 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.848+0000 7fb1f1b41700 1 --2- 172.21.15.195:0/4237971071 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb1d802ff60 0x7fb1d8032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fb1dc0096a0 tx=0x7fb1dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:48.860 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:48.852+0000 7fb1e2ffd700 1 -- 172.21.15.195:0/4237971071 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1e80079c0 con 0x7fb1ec0f6510 2023-10-12T22:36:49.058 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:49.056+0000 7fb1f45a6700 1 -- 172.21.15.195:0/4237971071 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1ec0fbf60 con 0x7fb1ec0f6510 2023-10-12T22:36:49.063 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:49.057+0000 7fb1e2ffd700 1 -- 172.21.15.195:0/4237971071 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb1e800f300 con 0x7fb1ec0f6510 2023-10-12T22:36:49.063 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:49.064 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:49.065 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:49.058+0000 7fb1f45a6700 1 -- 172.21.15.195:0/4237971071 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb1d802ff60 msgr2=0x7fb1d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:49.066 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:49.058+0000 7fb1f45a6700 1 --2- 172.21.15.195:0/4237971071 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb1d802ff60 0x7fb1d8032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fb1dc0096a0 tx=0x7fb1dc006b40).stop 2023-10-12T22:36:49.066 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:49.058+0000 7fb1f45a6700 1 -- 172.21.15.195:0/4237971071 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1ec0f6510 msgr2=0x7fb1ec106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:49.066 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:49.058+0000 7fb1f45a6700 1 --2- 172.21.15.195:0/4237971071 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1ec0f6510 0x7fb1ec106730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb1e8014000 tx=0x7fb1e800b040).stop 2023-10-12T22:36:49.066 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:49.058+0000 7fb1f45a6700 1 -- 172.21.15.195:0/4237971071 shutdown_connections 2023-10-12T22:36:49.067 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:49.058+0000 7fb1f45a6700 1 --2- 172.21.15.195:0/4237971071 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb1d802ff60 0x7fb1d8032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:49.067 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:49.058+0000 7fb1f45a6700 1 --2- 172.21.15.195:0/4237971071 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1ec0f6510 0x7fb1ec106730 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:49.067 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:49.058+0000 7fb1f45a6700 1 -- 172.21.15.195:0/4237971071 >> 172.21.15.195:0/4237971071 conn(0x7fb1ec0f1c70 msgr2=0x7fb1ec0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:49.067 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:49.059+0000 7fb1f45a6700 1 -- 172.21.15.195:0/4237971071 shutdown_connections 2023-10-12T22:36:49.067 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:49.059+0000 7fb1f45a6700 1 -- 172.21.15.195:0/4237971071 wait complete. 2023-10-12T22:36:49.068 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:49.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:49 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:49.058323+0000 mon.smithi131 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.195:0/4237971071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:50.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:50.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:50.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:48.847567+0000 mgr.smithi131.efktma (mgr.14162) 164 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:51.605 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.603+0000 7f05b307e700 1 -- 172.21.15.195:0/2779580063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05ac0f51a0 msgr2=0x7f05ac0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:51.605 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.603+0000 7f05b307e700 1 --2- 172.21.15.195:0/2779580063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05ac0f51a0 0x7f05ac0f55c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f05a8009fb0 tx=0x7f05a8009b30).stop 2023-10-12T22:36:51.605 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.603+0000 7f05b307e700 1 -- 172.21.15.195:0/2779580063 shutdown_connections 2023-10-12T22:36:51.605 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.603+0000 7f05b307e700 1 --2- 172.21.15.195:0/2779580063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05ac0f51a0 0x7f05ac0f55c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:51.606 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.603+0000 7f05b307e700 1 -- 172.21.15.195:0/2779580063 >> 172.21.15.195:0/2779580063 conn(0x7f05ac0f09a0 msgr2=0x7f05ac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:51.606 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.603+0000 7f05b307e700 1 -- 172.21.15.195:0/2779580063 shutdown_connections 2023-10-12T22:36:51.606 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.603+0000 7f05b307e700 1 -- 172.21.15.195:0/2779580063 wait complete. 2023-10-12T22:36:51.606 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.603+0000 7f05b307e700 1 Processor -- start 2023-10-12T22:36:51.606 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.603+0000 7f05b307e700 1 -- start start 2023-10-12T22:36:51.607 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.603+0000 7f05b307e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05ac0f51a0 0x7f05ac1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:51.607 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.603+0000 7f05b307e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac1069d0 con 0x7f05ac0f51a0 2023-10-12T22:36:51.607 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.604+0000 7f05b207c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05ac0f51a0 0x7f05ac1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:51.607 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.604+0000 7f05b207c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05ac0f51a0 0x7f05ac1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51014/0 (socket says 172.21.15.195:51014) 2023-10-12T22:36:51.607 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.604+0000 7f05b207c700 1 -- 172.21.15.195:0/2249670518 learned_addr learned my addr 172.21.15.195:0/2249670518 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:51.608 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.604+0000 7f05b207c700 1 -- 172.21.15.195:0/2249670518 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a8005040 con 0x7f05ac0f51a0 2023-10-12T22:36:51.609 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.604+0000 7f05b207c700 1 --2- 172.21.15.195:0/2249670518 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05ac0f51a0 0x7f05ac1064e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f05a8009830 tx=0x7f05a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:51.609 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.604+0000 7f05a2ffd700 1 -- 172.21.15.195:0/2249670518 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05a8009320 con 0x7f05ac0f51a0 2023-10-12T22:36:51.609 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.604+0000 7f05b307e700 1 -- 172.21.15.195:0/2249670518 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac106bd0 con 0x7f05ac0f51a0 2023-10-12T22:36:51.609 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.604+0000 7f05a2ffd700 1 -- 172.21.15.195:0/2249670518 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05a80136c0 con 0x7f05ac0f51a0 2023-10-12T22:36:51.610 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.604+0000 7f05a2ffd700 1 -- 172.21.15.195:0/2249670518 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05a801ee90 con 0x7f05ac0f51a0 2023-10-12T22:36:51.610 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.604+0000 7f05b307e700 1 -- 172.21.15.195:0/2249670518 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac107060 con 0x7f05ac0f51a0 2023-10-12T22:36:51.610 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.605+0000 7f05a2ffd700 1 -- 172.21.15.195:0/2249670518 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05a801e420 con 0x7f05ac0f51a0 2023-10-12T22:36:51.610 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.605+0000 7f05b307e700 1 -- 172.21.15.195:0/2249670518 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ac045ad0 con 0x7f05ac0f51a0 2023-10-12T22:36:51.611 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.605+0000 7f05a2ffd700 1 --2- 172.21.15.195:0/2249670518 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f059802ff10 0x7f05980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:51.611 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.605+0000 7f05a2ffd700 1 -- 172.21.15.195:0/2249670518 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f05a8040f40 con 0x7f05ac0f51a0 2023-10-12T22:36:51.611 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.606+0000 7f05b187b700 1 --2- 172.21.15.195:0/2249670518 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f059802ff10 0x7f05980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:51.611 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.606+0000 7f05b187b700 1 --2- 172.21.15.195:0/2249670518 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f059802ff10 0x7f05980323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f059c0096a0 tx=0x7f059c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:51.612 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.610+0000 7f05a2ffd700 1 -- 172.21.15.195:0/2249670518 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05a806d2b0 con 0x7f05ac0f51a0 2023-10-12T22:36:51.817 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.815+0000 7f05b307e700 1 -- 172.21.15.195:0/2249670518 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05ac107f10 con 0x7f05ac0f51a0 2023-10-12T22:36:51.818 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.816+0000 7f05a2ffd700 1 -- 172.21.15.195:0/2249670518 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f05a80095f0 con 0x7f05ac0f51a0 2023-10-12T22:36:51.818 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:51.819 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:51.819 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.818+0000 7f05b307e700 1 -- 172.21.15.195:0/2249670518 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f059802ff10 msgr2=0x7f05980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:51.820 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.818+0000 7f05b307e700 1 --2- 172.21.15.195:0/2249670518 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f059802ff10 0x7f05980323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f059c0096a0 tx=0x7f059c006b40).stop 2023-10-12T22:36:51.820 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.818+0000 7f05b307e700 1 -- 172.21.15.195:0/2249670518 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05ac0f51a0 msgr2=0x7f05ac1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:51.820 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.818+0000 7f05b307e700 1 --2- 172.21.15.195:0/2249670518 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05ac0f51a0 0x7f05ac1064e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f05a8009830 tx=0x7f05a800b040).stop 2023-10-12T22:36:51.820 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.818+0000 7f05b307e700 1 -- 172.21.15.195:0/2249670518 shutdown_connections 2023-10-12T22:36:51.820 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.818+0000 7f05b307e700 1 --2- 172.21.15.195:0/2249670518 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f059802ff10 0x7f05980323d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:51.821 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.818+0000 7f05b307e700 1 --2- 172.21.15.195:0/2249670518 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05ac0f51a0 0x7f05ac1064e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:51.821 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.818+0000 7f05b307e700 1 -- 172.21.15.195:0/2249670518 >> 172.21.15.195:0/2249670518 conn(0x7f05ac0f09a0 msgr2=0x7f05ac05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:51.821 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.818+0000 7f05b307e700 1 -- 172.21.15.195:0/2249670518 shutdown_connections 2023-10-12T22:36:51.821 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:51.818+0000 7f05b307e700 1 -- 172.21.15.195:0/2249670518 wait complete. 2023-10-12T22:36:51.821 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:52.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:50.847833+0000 mgr.smithi131.efktma (mgr.14162) 165 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:52.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:51.817533+0000 mon.smithi131 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.195:0/2249670518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:53.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:53.234 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:54.348 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.347+0000 7f019758b700 1 -- 172.21.15.195:0/3510703687 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f01900f6510 msgr2=0x7f01900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:54.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.347+0000 7f019758b700 1 --2- 172.21.15.195:0/3510703687 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f01900f6510 0x7f01900f6930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f0180009fb0 tx=0x7f0180009b30).stop 2023-10-12T22:36:54.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.347+0000 7f019758b700 1 -- 172.21.15.195:0/3510703687 shutdown_connections 2023-10-12T22:36:54.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.347+0000 7f019758b700 1 --2- 172.21.15.195:0/3510703687 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f01900f6510 0x7f01900f6930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:54.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.347+0000 7f019758b700 1 -- 172.21.15.195:0/3510703687 >> 172.21.15.195:0/3510703687 conn(0x7f01900f1c70 msgr2=0x7f01900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:54.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.347+0000 7f019758b700 1 -- 172.21.15.195:0/3510703687 shutdown_connections 2023-10-12T22:36:54.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.347+0000 7f019758b700 1 -- 172.21.15.195:0/3510703687 wait complete. 2023-10-12T22:36:54.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.348+0000 7f019758b700 1 Processor -- start 2023-10-12T22:36:54.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.348+0000 7f019758b700 1 -- start start 2023-10-12T22:36:54.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.348+0000 7f019758b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f01900f6510 0x7f0190106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:54.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.348+0000 7f019758b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0190106c20 con 0x7f01900f6510 2023-10-12T22:36:54.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.348+0000 7f0195327700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f01900f6510 0x7f0190106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:54.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.348+0000 7f0195327700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f01900f6510 0x7f0190106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51036/0 (socket says 172.21.15.195:51036) 2023-10-12T22:36:54.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.348+0000 7f0195327700 1 -- 172.21.15.195:0/2749015546 learned_addr learned my addr 172.21.15.195:0/2749015546 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:54.354 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.348+0000 7f0195327700 1 -- 172.21.15.195:0/2749015546 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0180005040 con 0x7f01900f6510 2023-10-12T22:36:54.354 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.349+0000 7f0195327700 1 --2- 172.21.15.195:0/2749015546 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f01900f6510 0x7f0190106730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f0180014000 tx=0x7f018000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:54.354 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.349+0000 7f0185ffb700 1 -- 172.21.15.195:0/2749015546 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01800095f0 con 0x7f01900f6510 2023-10-12T22:36:54.354 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.349+0000 7f019758b700 1 -- 172.21.15.195:0/2749015546 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0190106e20 con 0x7f01900f6510 2023-10-12T22:36:54.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.349+0000 7f0185ffb700 1 -- 172.21.15.195:0/2749015546 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0180007e70 con 0x7f01900f6510 2023-10-12T22:36:54.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.349+0000 7f0185ffb700 1 -- 172.21.15.195:0/2749015546 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0180014ea0 con 0x7f01900f6510 2023-10-12T22:36:54.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.349+0000 7f019758b700 1 -- 172.21.15.195:0/2749015546 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01901072b0 con 0x7f01900f6510 2023-10-12T22:36:54.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.350+0000 7f0185ffb700 1 -- 172.21.15.195:0/2749015546 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0180025070 con 0x7f01900f6510 2023-10-12T22:36:54.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.350+0000 7f0185ffb700 1 --2- 172.21.15.195:0/2749015546 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f017c02ff60 0x7f017c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:54.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.350+0000 7f019758b700 1 -- 172.21.15.195:0/2749015546 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0190045ad0 con 0x7f01900f6510 2023-10-12T22:36:54.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.350+0000 7f0185ffb700 1 -- 172.21.15.195:0/2749015546 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0180040990 con 0x7f01900f6510 2023-10-12T22:36:54.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.350+0000 7f0194b26700 1 --2- 172.21.15.195:0/2749015546 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f017c02ff60 0x7f017c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:54.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.351+0000 7f0194b26700 1 --2- 172.21.15.195:0/2749015546 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f017c02ff60 0x7f017c032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f018c0096a0 tx=0x7f018c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:54.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.355+0000 7f0185ffb700 1 -- 172.21.15.195:0/2749015546 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0180026350 con 0x7f01900f6510 2023-10-12T22:36:54.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:52.848259+0000 mgr.smithi131.efktma (mgr.14162) 166 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:54.567 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.566+0000 7f019758b700 1 -- 172.21.15.195:0/2749015546 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01900fc050 con 0x7f01900f6510 2023-10-12T22:36:54.568 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.567+0000 7f0185ffb700 1 -- 172.21.15.195:0/2749015546 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0180012330 con 0x7f01900f6510 2023-10-12T22:36:54.568 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:54.569 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:54.569 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.568+0000 7f019758b700 1 -- 172.21.15.195:0/2749015546 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f017c02ff60 msgr2=0x7f017c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:54.570 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.568+0000 7f019758b700 1 --2- 172.21.15.195:0/2749015546 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f017c02ff60 0x7f017c032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f018c0096a0 tx=0x7f018c006b40).stop 2023-10-12T22:36:54.570 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.568+0000 7f019758b700 1 -- 172.21.15.195:0/2749015546 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f01900f6510 msgr2=0x7f0190106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:54.570 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.568+0000 7f019758b700 1 --2- 172.21.15.195:0/2749015546 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f01900f6510 0x7f0190106730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f0180014000 tx=0x7f018000b040).stop 2023-10-12T22:36:54.570 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.568+0000 7f019758b700 1 -- 172.21.15.195:0/2749015546 shutdown_connections 2023-10-12T22:36:54.570 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.568+0000 7f019758b700 1 --2- 172.21.15.195:0/2749015546 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f017c02ff60 0x7f017c032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:54.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.568+0000 7f019758b700 1 --2- 172.21.15.195:0/2749015546 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f01900f6510 0x7f0190106730 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:54.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.568+0000 7f019758b700 1 -- 172.21.15.195:0/2749015546 >> 172.21.15.195:0/2749015546 conn(0x7f01900f1c70 msgr2=0x7f01900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:54.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.568+0000 7f019758b700 1 -- 172.21.15.195:0/2749015546 shutdown_connections 2023-10-12T22:36:54.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:54.568+0000 7f019758b700 1 -- 172.21.15.195:0/2749015546 wait complete. 2023-10-12T22:36:54.572 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:55.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:55 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:54.568026+0000 mon.smithi131 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.195:0/2749015546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:55.979 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:55.980 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:56.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:54.848674+0000 mgr.smithi131.efktma (mgr.14162) 167 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:57.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.043+0000 7f252a13d700 1 -- 172.21.15.195:0/292762303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25240f5410 msgr2=0x7f25240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:57.046 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.043+0000 7f252a13d700 1 --2- 172.21.15.195:0/292762303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25240f5410 0x7f25240f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f2514009fb0 tx=0x7f2514009b30).stop 2023-10-12T22:36:57.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.043+0000 7f252a13d700 1 -- 172.21.15.195:0/292762303 shutdown_connections 2023-10-12T22:36:57.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.043+0000 7f252a13d700 1 --2- 172.21.15.195:0/292762303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25240f5410 0x7f25240f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:57.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.043+0000 7f252a13d700 1 -- 172.21.15.195:0/292762303 >> 172.21.15.195:0/292762303 conn(0x7f25240f0b70 msgr2=0x7f25240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:57.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.043+0000 7f252a13d700 1 -- 172.21.15.195:0/292762303 shutdown_connections 2023-10-12T22:36:57.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.043+0000 7f252a13d700 1 -- 172.21.15.195:0/292762303 wait complete. 2023-10-12T22:36:57.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.044+0000 7f252a13d700 1 Processor -- start 2023-10-12T22:36:57.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.044+0000 7f252a13d700 1 -- start start 2023-10-12T22:36:57.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.044+0000 7f252a13d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25240f5410 0x7f2524106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:57.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.044+0000 7f252a13d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2524106c10 con 0x7f25240f5410 2023-10-12T22:36:57.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.044+0000 7f25237fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25240f5410 0x7f2524106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:57.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.044+0000 7f25237fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25240f5410 0x7f2524106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51048/0 (socket says 172.21.15.195:51048) 2023-10-12T22:36:57.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.044+0000 7f25237fe700 1 -- 172.21.15.195:0/3187782804 learned_addr learned my addr 172.21.15.195:0/3187782804 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:57.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.045+0000 7f25237fe700 1 -- 172.21.15.195:0/3187782804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2514005040 con 0x7f25240f5410 2023-10-12T22:36:57.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.045+0000 7f25237fe700 1 --2- 172.21.15.195:0/3187782804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25240f5410 0x7f2524106720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2514013510 tx=0x7f251400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:57.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.045+0000 7f250bfff700 1 -- 172.21.15.195:0/3187782804 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2514009320 con 0x7f25240f5410 2023-10-12T22:36:57.049 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.045+0000 7f252a13d700 1 -- 172.21.15.195:0/3187782804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2524106e10 con 0x7f25240f5410 2023-10-12T22:36:57.050 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.045+0000 7f250bfff700 1 -- 172.21.15.195:0/3187782804 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2514013f00 con 0x7f25240f5410 2023-10-12T22:36:57.050 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.045+0000 7f250bfff700 1 -- 172.21.15.195:0/3187782804 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2514014d90 con 0x7f25240f5410 2023-10-12T22:36:57.050 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.045+0000 7f252a13d700 1 -- 172.21.15.195:0/3187782804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25241072a0 con 0x7f25240f5410 2023-10-12T22:36:57.051 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.046+0000 7f250bfff700 1 -- 172.21.15.195:0/3187782804 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25140095f0 con 0x7f25240f5410 2023-10-12T22:36:57.051 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.046+0000 7f252a13d700 1 -- 172.21.15.195:0/3187782804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2524045ad0 con 0x7f25240f5410 2023-10-12T22:36:57.051 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.046+0000 7f250bfff700 1 --2- 172.21.15.195:0/3187782804 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f250c02ff60 0x7f250c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:57.052 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.046+0000 7f250bfff700 1 -- 172.21.15.195:0/3187782804 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25140410a0 con 0x7f25240f5410 2023-10-12T22:36:57.052 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.046+0000 7f2522ffd700 1 --2- 172.21.15.195:0/3187782804 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f250c02ff60 0x7f250c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:57.052 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.047+0000 7f2522ffd700 1 --2- 172.21.15.195:0/3187782804 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f250c02ff60 0x7f250c032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f25180096a0 tx=0x7f2518006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:57.053 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.051+0000 7f250bfff700 1 -- 172.21.15.195:0/3187782804 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f251401d7f0 con 0x7f25240f5410 2023-10-12T22:36:57.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.257+0000 7f252a13d700 1 -- 172.21.15.195:0/3187782804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2524108150 con 0x7f25240f5410 2023-10-12T22:36:57.258 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.257+0000 7f250bfff700 1 -- 172.21.15.195:0/3187782804 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2514010230 con 0x7f25240f5410 2023-10-12T22:36:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:36:57.259 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:36:57.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.259+0000 7f252a13d700 1 -- 172.21.15.195:0/3187782804 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f250c02ff60 msgr2=0x7f250c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:57.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.259+0000 7f252a13d700 1 --2- 172.21.15.195:0/3187782804 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f250c02ff60 0x7f250c032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f25180096a0 tx=0x7f2518006b40).stop 2023-10-12T22:36:57.260 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.259+0000 7f252a13d700 1 -- 172.21.15.195:0/3187782804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25240f5410 msgr2=0x7f2524106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:57.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.259+0000 7f252a13d700 1 --2- 172.21.15.195:0/3187782804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25240f5410 0x7f2524106720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2514013510 tx=0x7f251400b040).stop 2023-10-12T22:36:57.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.259+0000 7f252a13d700 1 -- 172.21.15.195:0/3187782804 shutdown_connections 2023-10-12T22:36:57.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.259+0000 7f252a13d700 1 --2- 172.21.15.195:0/3187782804 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f250c02ff60 0x7f250c032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:57.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.259+0000 7f252a13d700 1 --2- 172.21.15.195:0/3187782804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25240f5410 0x7f2524106720 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:57.261 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.259+0000 7f252a13d700 1 -- 172.21.15.195:0/3187782804 >> 172.21.15.195:0/3187782804 conn(0x7f25240f0b70 msgr2=0x7f252405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:57.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.259+0000 7f252a13d700 1 -- 172.21.15.195:0/3187782804 shutdown_connections 2023-10-12T22:36:57.262 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:57.259+0000 7f252a13d700 1 -- 172.21.15.195:0/3187782804 wait complete. 2023-10-12T22:36:57.263 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:36:58.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:58 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:56.848950+0000 mgr.smithi131.efktma (mgr.14162) 168 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:36:58.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:36:58 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:36:57.258710+0000 mon.smithi131 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.195:0/3187782804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:36:58.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:36:58.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:36:59.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.874+0000 7efe9c98b700 1 -- 172.21.15.195:0/3000691666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efe980f51a0 msgr2=0x7efe980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:36:59.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.874+0000 7efe9c98b700 1 --2- 172.21.15.195:0/3000691666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efe980f51a0 0x7efe980f55c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7efe88009fb0 tx=0x7efe88009b30).stop 2023-10-12T22:36:59.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.874+0000 7efe9c98b700 1 -- 172.21.15.195:0/3000691666 shutdown_connections 2023-10-12T22:36:59.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.874+0000 7efe9c98b700 1 --2- 172.21.15.195:0/3000691666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efe980f51a0 0x7efe980f55c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:36:59.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.874+0000 7efe9c98b700 1 -- 172.21.15.195:0/3000691666 >> 172.21.15.195:0/3000691666 conn(0x7efe980f09a0 msgr2=0x7efe980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:36:59.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.874+0000 7efe9c98b700 1 -- 172.21.15.195:0/3000691666 shutdown_connections 2023-10-12T22:36:59.876 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.874+0000 7efe9c98b700 1 -- 172.21.15.195:0/3000691666 wait complete. 2023-10-12T22:36:59.879 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.874+0000 7efe9c98b700 1 Processor -- start 2023-10-12T22:36:59.879 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.875+0000 7efe9c98b700 1 -- start start 2023-10-12T22:36:59.879 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.875+0000 7efe9c98b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efe981065f0 0x7efe98106a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:59.879 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.875+0000 7efe9c98b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe98106f00 con 0x7efe981065f0 2023-10-12T22:36:59.879 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.875+0000 7efe96d9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efe981065f0 0x7efe98106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:59.880 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.875+0000 7efe96d9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efe981065f0 0x7efe98106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:35022/0 (socket says 172.21.15.195:35022) 2023-10-12T22:36:59.880 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.875+0000 7efe96d9d700 1 -- 172.21.15.195:0/1635955622 learned_addr learned my addr 172.21.15.195:0/1635955622 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:36:59.880 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.875+0000 7efe96d9d700 1 -- 172.21.15.195:0/1635955622 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe88005040 con 0x7efe981065f0 2023-10-12T22:36:59.880 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.875+0000 7efe96d9d700 1 --2- 172.21.15.195:0/1635955622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efe981065f0 0x7efe98106a10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7efe88009f80 tx=0x7efe88009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:59.880 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.876+0000 7efe7f7fe700 1 -- 172.21.15.195:0/1635955622 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe880136c0 con 0x7efe981065f0 2023-10-12T22:36:59.881 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.876+0000 7efe9c98b700 1 -- 172.21.15.195:0/1635955622 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe98107100 con 0x7efe981065f0 2023-10-12T22:36:59.881 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.876+0000 7efe7f7fe700 1 -- 172.21.15.195:0/1635955622 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe880095f0 con 0x7efe981065f0 2023-10-12T22:36:59.881 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.876+0000 7efe7f7fe700 1 -- 172.21.15.195:0/1635955622 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe88014420 con 0x7efe981065f0 2023-10-12T22:36:59.881 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.876+0000 7efe9c98b700 1 -- 172.21.15.195:0/1635955622 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe98107c50 con 0x7efe981065f0 2023-10-12T22:36:59.881 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.876+0000 7efe7f7fe700 1 -- 172.21.15.195:0/1635955622 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe88025060 con 0x7efe981065f0 2023-10-12T22:36:59.882 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.877+0000 7efe7f7fe700 1 --2- 172.21.15.195:0/1635955622 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7efe8002ff60 0x7efe80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:36:59.882 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.877+0000 7efe7f7fe700 1 -- 172.21.15.195:0/1635955622 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe88040da0 con 0x7efe981065f0 2023-10-12T22:36:59.882 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.877+0000 7efe9c98b700 1 -- 172.21.15.195:0/1635955622 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe98059100 con 0x7efe981065f0 2023-10-12T22:36:59.882 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.877+0000 7efe9659c700 1 --2- 172.21.15.195:0/1635955622 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7efe8002ff60 0x7efe80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:36:59.882 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.877+0000 7efe9659c700 1 --2- 172.21.15.195:0/1635955622 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7efe8002ff60 0x7efe80032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7efe8c0096a0 tx=0x7efe8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:36:59.883 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:36:59.881+0000 7efe7f7fe700 1 -- 172.21.15.195:0/1635955622 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe88004490 con 0x7efe981065f0 2023-10-12T22:37:00.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:00.087+0000 7efe9c98b700 1 -- 172.21.15.195:0/1635955622 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe98108410 con 0x7efe981065f0 2023-10-12T22:37:00.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:00.087+0000 7efe7f7fe700 1 -- 172.21.15.195:0/1635955622 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efe88004490 con 0x7efe981065f0 2023-10-12T22:37:00.090 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:00.090 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:00.091 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:00.089+0000 7efe9c98b700 1 -- 172.21.15.195:0/1635955622 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7efe8002ff60 msgr2=0x7efe80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:00.092 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:00.089+0000 7efe9c98b700 1 --2- 172.21.15.195:0/1635955622 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7efe8002ff60 0x7efe80032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7efe8c0096a0 tx=0x7efe8c006b40).stop 2023-10-12T22:37:00.092 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:00.089+0000 7efe9c98b700 1 -- 172.21.15.195:0/1635955622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efe981065f0 msgr2=0x7efe98106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:00.092 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:00.089+0000 7efe9c98b700 1 --2- 172.21.15.195:0/1635955622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efe981065f0 0x7efe98106a10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7efe88009f80 tx=0x7efe88009b30).stop 2023-10-12T22:37:00.092 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:00.089+0000 7efe9c98b700 1 -- 172.21.15.195:0/1635955622 shutdown_connections 2023-10-12T22:37:00.093 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:00.089+0000 7efe9c98b700 1 --2- 172.21.15.195:0/1635955622 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7efe8002ff60 0x7efe80032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:00.093 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:00.089+0000 7efe9c98b700 1 --2- 172.21.15.195:0/1635955622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efe981065f0 0x7efe98106a10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:00.093 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:00.089+0000 7efe9c98b700 1 -- 172.21.15.195:0/1635955622 >> 172.21.15.195:0/1635955622 conn(0x7efe980f09a0 msgr2=0x7efe9805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:00.094 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:00.089+0000 7efe9c98b700 1 -- 172.21.15.195:0/1635955622 shutdown_connections 2023-10-12T22:37:00.094 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:00.089+0000 7efe9c98b700 1 -- 172.21.15.195:0/1635955622 wait complete. 2023-10-12T22:37:00.094 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:00.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:36:58.849206+0000 mgr.smithi131.efktma (mgr.14162) 169 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:00.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:00.088781+0000 mon.smithi131 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.195:0/1635955622' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:01.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:01.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:02.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:02 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:00.849454+0000 mgr.smithi131.efktma (mgr.14162) 170 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:02.680 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.678+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3261991054 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f3c0601e0 msgr2=0x7f2f3c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:02.680 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.678+0000 7f2f41cc1700 1 --2- 172.21.15.195:0/3261991054 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f3c0601e0 0x7f2f3c0f9190 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f2f2c009fb0 tx=0x7f2f2c009b30).stop 2023-10-12T22:37:02.681 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.678+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3261991054 shutdown_connections 2023-10-12T22:37:02.681 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.678+0000 7f2f41cc1700 1 --2- 172.21.15.195:0/3261991054 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f3c0601e0 0x7f2f3c0f9190 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:02.681 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.678+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3261991054 >> 172.21.15.195:0/3261991054 conn(0x7f2f3c0f0b70 msgr2=0x7f2f3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:02.681 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.678+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3261991054 shutdown_connections 2023-10-12T22:37:02.681 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.679+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3261991054 wait complete. 2023-10-12T22:37:02.682 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.679+0000 7f2f41cc1700 1 Processor -- start 2023-10-12T22:37:02.683 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.679+0000 7f2f41cc1700 1 -- start start 2023-10-12T22:37:02.684 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.680+0000 7f2f41cc1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f3c0601e0 0x7f2f3c1066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:02.684 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.680+0000 7f2f41cc1700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c106ba0 con 0x7f2f3c0601e0 2023-10-12T22:37:02.684 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.680+0000 7f2f3b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f3c0601e0 0x7f2f3c1066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:02.684 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.680+0000 7f2f3b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f3c0601e0 0x7f2f3c1066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:35038/0 (socket says 172.21.15.195:35038) 2023-10-12T22:37:02.684 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.680+0000 7f2f3b7fe700 1 -- 172.21.15.195:0/3678695994 learned_addr learned my addr 172.21.15.195:0/3678695994 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:02.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.680+0000 7f2f3b7fe700 1 -- 172.21.15.195:0/3678695994 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c005040 con 0x7f2f3c0601e0 2023-10-12T22:37:02.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.680+0000 7f2f3b7fe700 1 --2- 172.21.15.195:0/3678695994 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f3c0601e0 0x7f2f3c1066b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f2f2c004dd0 tx=0x7f2f2c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:02.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.681+0000 7f2f33fff700 1 -- 172.21.15.195:0/3678695994 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f2c018070 con 0x7f2f3c0601e0 2023-10-12T22:37:02.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.681+0000 7f2f33fff700 1 -- 172.21.15.195:0/3678695994 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f2c013410 con 0x7f2f3c0601e0 2023-10-12T22:37:02.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.681+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3678695994 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f3c106da0 con 0x7f2f3c0601e0 2023-10-12T22:37:02.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.681+0000 7f2f33fff700 1 -- 172.21.15.195:0/3678695994 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f2c01c3c0 con 0x7f2f3c0601e0 2023-10-12T22:37:02.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.681+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3678695994 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3c107230 con 0x7f2f3c0601e0 2023-10-12T22:37:02.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.681+0000 7f2f33fff700 1 -- 172.21.15.195:0/3678695994 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f2c01c630 con 0x7f2f3c0601e0 2023-10-12T22:37:02.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.682+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3678695994 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3c045ad0 con 0x7f2f3c0601e0 2023-10-12T22:37:02.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.682+0000 7f2f33fff700 1 --2- 172.21.15.195:0/3678695994 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2f2802ff60 0x7f2f28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:02.687 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.682+0000 7f2f33fff700 1 -- 172.21.15.195:0/3678695994 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f2c041390 con 0x7f2f3c0601e0 2023-10-12T22:37:02.687 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.682+0000 7f2f3affd700 1 --2- 172.21.15.195:0/3678695994 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2f2802ff60 0x7f2f28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:02.687 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.682+0000 7f2f3affd700 1 --2- 172.21.15.195:0/3678695994 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2f2802ff60 0x7f2f28032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f2f240096a0 tx=0x7f2f24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:02.688 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.687+0000 7f2f33fff700 1 -- 172.21.15.195:0/3678695994 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f2c027960 con 0x7f2f3c0601e0 2023-10-12T22:37:02.895 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.894+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3678695994 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f3c108080 con 0x7f2f3c0601e0 2023-10-12T22:37:02.896 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.895+0000 7f2f33fff700 1 -- 172.21.15.195:0/3678695994 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2f2c01b9c0 con 0x7f2f3c0601e0 2023-10-12T22:37:02.896 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:02.897 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:02.898 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.896+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3678695994 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2f2802ff60 msgr2=0x7f2f28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:02.898 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.896+0000 7f2f41cc1700 1 --2- 172.21.15.195:0/3678695994 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2f2802ff60 0x7f2f28032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f2f240096a0 tx=0x7f2f24006b40).stop 2023-10-12T22:37:02.898 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.896+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3678695994 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f3c0601e0 msgr2=0x7f2f3c1066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:02.898 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.896+0000 7f2f41cc1700 1 --2- 172.21.15.195:0/3678695994 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f3c0601e0 0x7f2f3c1066b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f2f2c004dd0 tx=0x7f2f2c009b30).stop 2023-10-12T22:37:02.898 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.896+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3678695994 shutdown_connections 2023-10-12T22:37:02.899 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.896+0000 7f2f41cc1700 1 --2- 172.21.15.195:0/3678695994 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2f2802ff60 0x7f2f28032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:02.899 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.896+0000 7f2f41cc1700 1 --2- 172.21.15.195:0/3678695994 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f3c0601e0 0x7f2f3c1066b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:02.899 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.896+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3678695994 >> 172.21.15.195:0/3678695994 conn(0x7f2f3c0f0b70 msgr2=0x7f2f3c0f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:02.900 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.896+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3678695994 shutdown_connections 2023-10-12T22:37:02.900 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:02.896+0000 7f2f41cc1700 1 -- 172.21.15.195:0/3678695994 wait complete. 2023-10-12T22:37:02.901 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:03.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:03 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:02.896210+0000 mon.smithi131 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.195:0/3678695994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:04.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:04.353 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:04.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:02.849709+0000 mgr.smithi131.efktma (mgr.14162) 171 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:05.505 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.502+0000 7f329626e700 1 -- 172.21.15.195:0/2510685867 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32900f5410 msgr2=0x7f32900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:05.506 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.502+0000 7f329626e700 1 --2- 172.21.15.195:0/2510685867 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32900f5410 0x7f32900f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f3280009fb0 tx=0x7f3280009b30).stop 2023-10-12T22:37:05.506 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.502+0000 7f329626e700 1 -- 172.21.15.195:0/2510685867 shutdown_connections 2023-10-12T22:37:05.506 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.502+0000 7f329626e700 1 --2- 172.21.15.195:0/2510685867 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32900f5410 0x7f32900f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:05.506 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.502+0000 7f329626e700 1 -- 172.21.15.195:0/2510685867 >> 172.21.15.195:0/2510685867 conn(0x7f32900f0b70 msgr2=0x7f32900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:05.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.502+0000 7f329626e700 1 -- 172.21.15.195:0/2510685867 shutdown_connections 2023-10-12T22:37:05.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.502+0000 7f329626e700 1 -- 172.21.15.195:0/2510685867 wait complete. 2023-10-12T22:37:05.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.503+0000 7f329626e700 1 Processor -- start 2023-10-12T22:37:05.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.503+0000 7f329626e700 1 -- start start 2023-10-12T22:37:05.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.503+0000 7f329626e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32900f5410 0x7f3290109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:05.508 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.503+0000 7f329626e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3290060220 con 0x7f32900f5410 2023-10-12T22:37:05.508 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.504+0000 7f328ffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32900f5410 0x7f3290109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:05.508 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.504+0000 7f328ffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32900f5410 0x7f3290109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:35056/0 (socket says 172.21.15.195:35056) 2023-10-12T22:37:05.508 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.504+0000 7f328ffff700 1 -- 172.21.15.195:0/854242111 learned_addr learned my addr 172.21.15.195:0/854242111 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:05.508 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.504+0000 7f328ffff700 1 -- 172.21.15.195:0/854242111 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3280005040 con 0x7f32900f5410 2023-10-12T22:37:05.508 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.504+0000 7f328ffff700 1 --2- 172.21.15.195:0/854242111 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32900f5410 0x7f3290109450 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f3280013510 tx=0x7f328000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:05.509 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.504+0000 7f328cff9700 1 -- 172.21.15.195:0/854242111 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3280009320 con 0x7f32900f5410 2023-10-12T22:37:05.509 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.504+0000 7f329626e700 1 -- 172.21.15.195:0/854242111 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3290060420 con 0x7f32900f5410 2023-10-12T22:37:05.509 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.504+0000 7f328cff9700 1 -- 172.21.15.195:0/854242111 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3280013f00 con 0x7f32900f5410 2023-10-12T22:37:05.509 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.504+0000 7f328cff9700 1 -- 172.21.15.195:0/854242111 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3280014d90 con 0x7f32900f5410 2023-10-12T22:37:05.509 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.504+0000 7f329626e700 1 -- 172.21.15.195:0/854242111 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3290109b00 con 0x7f32900f5410 2023-10-12T22:37:05.510 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.505+0000 7f328cff9700 1 -- 172.21.15.195:0/854242111 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3280014420 con 0x7f32900f5410 2023-10-12T22:37:05.511 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.505+0000 7f329626e700 1 -- 172.21.15.195:0/854242111 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3290059100 con 0x7f32900f5410 2023-10-12T22:37:05.511 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.505+0000 7f328cff9700 1 --2- 172.21.15.195:0/854242111 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f327802ff60 0x7f3278032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:05.511 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.505+0000 7f328cff9700 1 -- 172.21.15.195:0/854242111 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3280041000 con 0x7f32900f5410 2023-10-12T22:37:05.511 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.506+0000 7f328f7fe700 1 --2- 172.21.15.195:0/854242111 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f327802ff60 0x7f3278032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:05.511 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.506+0000 7f328f7fe700 1 --2- 172.21.15.195:0/854242111 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f327802ff60 0x7f3278032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f32840096a0 tx=0x7f3284006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:05.512 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.510+0000 7f328cff9700 1 -- 172.21.15.195:0/854242111 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32800095f0 con 0x7f32900f5410 2023-10-12T22:37:05.718 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.717+0000 7f329626e700 1 -- 172.21.15.195:0/854242111 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3290045ad0 con 0x7f32900f5410 2023-10-12T22:37:05.719 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.718+0000 7f328cff9700 1 -- 172.21.15.195:0/854242111 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f328001db70 con 0x7f32900f5410 2023-10-12T22:37:05.719 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:05.720 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:05.721 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.720+0000 7f329626e700 1 -- 172.21.15.195:0/854242111 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f327802ff60 msgr2=0x7f3278032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:05.721 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.720+0000 7f329626e700 1 --2- 172.21.15.195:0/854242111 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f327802ff60 0x7f3278032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f32840096a0 tx=0x7f3284006b40).stop 2023-10-12T22:37:05.721 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.720+0000 7f329626e700 1 -- 172.21.15.195:0/854242111 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32900f5410 msgr2=0x7f3290109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:05.722 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.720+0000 7f329626e700 1 --2- 172.21.15.195:0/854242111 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32900f5410 0x7f3290109450 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f3280013510 tx=0x7f328000b040).stop 2023-10-12T22:37:05.722 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.720+0000 7f329626e700 1 -- 172.21.15.195:0/854242111 shutdown_connections 2023-10-12T22:37:05.722 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.720+0000 7f329626e700 1 --2- 172.21.15.195:0/854242111 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f327802ff60 0x7f3278032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:05.722 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.720+0000 7f329626e700 1 --2- 172.21.15.195:0/854242111 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32900f5410 0x7f3290109450 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:05.722 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.720+0000 7f329626e700 1 -- 172.21.15.195:0/854242111 >> 172.21.15.195:0/854242111 conn(0x7f32900f0b70 msgr2=0x7f32900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:05.723 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.720+0000 7f329626e700 1 -- 172.21.15.195:0/854242111 shutdown_connections 2023-10-12T22:37:05.723 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:05.720+0000 7f329626e700 1 -- 172.21.15.195:0/854242111 wait complete. 2023-10-12T22:37:05.723 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:06.229 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:04.850038+0000 mgr.smithi131.efktma (mgr.14162) 172 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:06.229 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:05.719479+0000 mon.smithi131 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.195:0/854242111' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:06.229 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:05.893866+0000 mon.smithi131 (mon.0) 240 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-10-12T22:37:06.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:05.896299+0000 mon.smithi131 (mon.0) 241 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-10-12T22:37:06.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:05.897108+0000 mon.smithi131 (mon.0) 242 : audit [DBG] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-12T22:37:07.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:07.144 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:07.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:37:05.892352+0000 mgr.smithi131.efktma (mgr.14162) 173 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi131 ... 2023-10-12T22:37:07.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-10-12T22:37:07.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-10-12T22:37:07.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr time="2023-10-12T22:32:08Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-10-12T22:37:07.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr time="2023-10-12T22:32:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-10-12T22:37:07.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr time="2023-10-12T22:33:23Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-10-12T22:37:07.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-10-12T22:37:07.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-10-12T22:37:07.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-10-12T22:37:07.175 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr time="2023-10-12T22:34:43Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-10-12T22:37:07.175 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr time="2023-10-12T22:35:20Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-10-12T22:37:07.175 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr time="2023-10-12T22:36:28Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-10-12T22:37:07.175 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-10-12T22:37:07.175 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Traceback (most recent call last): 2023-10-12T22:37:07.175 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-10-12T22:37:07.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: main() 2023-10-12T22:37:07.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-10-12T22:37:07.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: r = ctx.func(ctx) 2023-10-12T22:37:07.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-10-12T22:37:07.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: return func(ctx) 2023-10-12T22:37:07.177 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-10-12T22:37:07.177 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-10-12T22:37:07.177 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-10-12T22:37:07.177 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-10-12T22:37:07.177 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-10-12T22:37:07.178 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: raise RuntimeError('uid/gid not found') 2023-10-12T22:37:07.178 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: RuntimeError: uid/gid not found 2023-10-12T22:37:07.178 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Traceback (most recent call last): 2023-10-12T22:37:07.178 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-10-12T22:37:07.178 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: yield (conn, connr) 2023-10-12T22:37:07.179 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-10-12T22:37:07.179 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: code, '\n'.join(err))) 2023-10-12T22:37:07.179 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi131 ... 2023-10-12T22:37:07.179 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-10-12T22:37:07.179 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-10-12T22:37:07.179 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr time="2023-10-12T22:32:08Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-10-12T22:37:07.180 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr time="2023-10-12T22:32:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-10-12T22:37:07.180 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr time="2023-10-12T22:33:23Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-10-12T22:37:07.180 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-10-12T22:37:07.180 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-10-12T22:37:07.180 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-10-12T22:37:07.181 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr time="2023-10-12T22:34:43Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-10-12T22:37:07.181 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr time="2023-10-12T22:35:20Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-10-12T22:37:07.181 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr time="2023-10-12T22:36:28Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-10-12T22:37:07.181 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-10-12T22:37:07.182 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Traceback (most recent call last): 2023-10-12T22:37:07.182 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-10-12T22:37:07.182 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: main() 2023-10-12T22:37:07.182 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-10-12T22:37:07.182 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: r = ctx.func(ctx) 2023-10-12T22:37:07.183 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-10-12T22:37:07.183 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: return func(ctx) 2023-10-12T22:37:07.183 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-10-12T22:37:07.183 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-10-12T22:37:07.183 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-10-12T22:37:07.184 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-10-12T22:37:07.184 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-10-12T22:37:07.184 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: raise RuntimeError('uid/gid not found') 2023-10-12T22:37:07.184 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: RuntimeError: uid/gid not found 2023-10-12T22:37:07.184 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:07 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:37:05.897693+0000 mgr.smithi131.efktma (mgr.14162) 174 : cephadm [INF] Deploying daemon crash.smithi131 on smithi131 2023-10-12T22:37:08.278 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.277+0000 7feea5ff5700 1 -- 172.21.15.195:0/2427916623 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea00f51a0 msgr2=0x7feea00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:08.279 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.277+0000 7feea5ff5700 1 --2- 172.21.15.195:0/2427916623 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea00f51a0 0x7feea00f55c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fee94009fb0 tx=0x7fee94009b30).stop 2023-10-12T22:37:08.279 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.277+0000 7feea5ff5700 1 -- 172.21.15.195:0/2427916623 shutdown_connections 2023-10-12T22:37:08.279 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.277+0000 7feea5ff5700 1 --2- 172.21.15.195:0/2427916623 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea00f51a0 0x7feea00f55c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:08.279 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.277+0000 7feea5ff5700 1 -- 172.21.15.195:0/2427916623 >> 172.21.15.195:0/2427916623 conn(0x7feea00f09a0 msgr2=0x7feea00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:08.280 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.277+0000 7feea5ff5700 1 -- 172.21.15.195:0/2427916623 shutdown_connections 2023-10-12T22:37:08.280 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.277+0000 7feea5ff5700 1 -- 172.21.15.195:0/2427916623 wait complete. 2023-10-12T22:37:08.280 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.277+0000 7feea5ff5700 1 Processor -- start 2023-10-12T22:37:08.280 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.277+0000 7feea5ff5700 1 -- start start 2023-10-12T22:37:08.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.278+0000 7feea5ff5700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea00f51a0 0x7feea01091a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:08.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.278+0000 7feea5ff5700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea0060280 con 0x7feea00f51a0 2023-10-12T22:37:08.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.278+0000 7feea4ff3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea00f51a0 0x7feea01091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:08.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.278+0000 7feea4ff3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea00f51a0 0x7feea01091a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:49440/0 (socket says 172.21.15.195:49440) 2023-10-12T22:37:08.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.278+0000 7feea4ff3700 1 -- 172.21.15.195:0/3497260440 learned_addr learned my addr 172.21.15.195:0/3497260440 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:08.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.278+0000 7feea4ff3700 1 -- 172.21.15.195:0/3497260440 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee94005040 con 0x7feea00f51a0 2023-10-12T22:37:08.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.278+0000 7feea4ff3700 1 --2- 172.21.15.195:0/3497260440 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea00f51a0 0x7feea01091a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fee94013510 tx=0x7fee9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:08.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.279+0000 7fee9d7fa700 1 -- 172.21.15.195:0/3497260440 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee94009320 con 0x7feea00f51a0 2023-10-12T22:37:08.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.279+0000 7fee9d7fa700 1 -- 172.21.15.195:0/3497260440 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee94013f00 con 0x7feea00f51a0 2023-10-12T22:37:08.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.279+0000 7feea5ff5700 1 -- 172.21.15.195:0/3497260440 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea0060480 con 0x7feea00f51a0 2023-10-12T22:37:08.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.279+0000 7fee9d7fa700 1 -- 172.21.15.195:0/3497260440 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee94014dc0 con 0x7feea00f51a0 2023-10-12T22:37:08.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.279+0000 7feea5ff5700 1 -- 172.21.15.195:0/3497260440 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea01098d0 con 0x7feea00f51a0 2023-10-12T22:37:08.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.280+0000 7feea5ff5700 1 -- 172.21.15.195:0/3497260440 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea0045ad0 con 0x7feea00f51a0 2023-10-12T22:37:08.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.280+0000 7fee9d7fa700 1 -- 172.21.15.195:0/3497260440 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee940095f0 con 0x7feea00f51a0 2023-10-12T22:37:08.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.280+0000 7fee9d7fa700 1 --2- 172.21.15.195:0/3497260440 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fee9002ffb0 0x7fee90032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:08.286 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.280+0000 7fee9d7fa700 1 -- 172.21.15.195:0/3497260440 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee94040ff0 con 0x7feea00f51a0 2023-10-12T22:37:08.286 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.280+0000 7fee9ffff700 1 --2- 172.21.15.195:0/3497260440 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fee9002ffb0 0x7fee90032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:08.286 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.280+0000 7fee9ffff700 1 --2- 172.21.15.195:0/3497260440 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fee9002ffb0 0x7fee90032470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fee8c0096a0 tx=0x7fee8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:08.286 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.285+0000 7fee9d7fa700 1 -- 172.21.15.195:0/3497260440 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee9406d2b0 con 0x7feea00f51a0 2023-10-12T22:37:08.492 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.491+0000 7feea5ff5700 1 -- 172.21.15.195:0/3497260440 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feea00fabe0 con 0x7feea00f51a0 2023-10-12T22:37:08.493 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.492+0000 7fee9d7fa700 1 -- 172.21.15.195:0/3497260440 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fee94014420 con 0x7feea00f51a0 2023-10-12T22:37:08.493 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:08.494 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:08.495 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.494+0000 7feea5ff5700 1 -- 172.21.15.195:0/3497260440 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fee9002ffb0 msgr2=0x7fee90032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:08.495 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.494+0000 7feea5ff5700 1 --2- 172.21.15.195:0/3497260440 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fee9002ffb0 0x7fee90032470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fee8c0096a0 tx=0x7fee8c006b40).stop 2023-10-12T22:37:08.495 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.494+0000 7feea5ff5700 1 -- 172.21.15.195:0/3497260440 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea00f51a0 msgr2=0x7feea01091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:08.495 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.494+0000 7feea5ff5700 1 --2- 172.21.15.195:0/3497260440 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea00f51a0 0x7feea01091a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fee94013510 tx=0x7fee9400b040).stop 2023-10-12T22:37:08.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.494+0000 7feea5ff5700 1 -- 172.21.15.195:0/3497260440 shutdown_connections 2023-10-12T22:37:08.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.494+0000 7feea5ff5700 1 --2- 172.21.15.195:0/3497260440 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fee9002ffb0 0x7fee90032470 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:08.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.494+0000 7feea5ff5700 1 --2- 172.21.15.195:0/3497260440 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea00f51a0 0x7feea01091a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:08.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.494+0000 7feea5ff5700 1 -- 172.21.15.195:0/3497260440 >> 172.21.15.195:0/3497260440 conn(0x7feea00f09a0 msgr2=0x7feea00f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:08.496 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.494+0000 7feea5ff5700 1 -- 172.21.15.195:0/3497260440 shutdown_connections 2023-10-12T22:37:08.497 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:08.494+0000 7feea5ff5700 1 -- 172.21.15.195:0/3497260440 wait complete. 2023-10-12T22:37:08.497 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:08.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:06.850288+0000 mgr.smithi131.efktma (mgr.14162) 175 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:08.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:07.322307+0000 mon.smithi131 (mon.0) 243 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:37:08.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:07.524820+0000 mon.smithi131 (mon.0) 244 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:37:08.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:07.527929+0000 mon.smithi131 (mon.0) 245 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:37:08.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:07.528305+0000 mon.smithi131 (mon.0) 246 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-10-12T22:37:08.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:07.528726+0000 mgr.smithi131.efktma (mgr.14162) 176 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-10-12T22:37:08.741 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:07.531799+0000 mon.smithi131 (mon.0) 247 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:37:08.741 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:37:07.534994+0000 mgr.smithi131.efktma (mgr.14162) 177 : cephadm [INF] Deploying daemon grafana.smithi131 on smithi131 2023-10-12T22:37:09.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:09 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:08.493335+0000 mon.smithi131 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.195:0/3497260440' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:09.856 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:09.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:10.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:08.850572+0000 mgr.smithi131.efktma (mgr.14162) 178 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:10.980 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.980+0000 7f9b49600700 1 -- 172.21.15.195:0/1334238275 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b440f62c0 msgr2=0x7f9b440f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:10.980 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.980+0000 7f9b49600700 1 --2- 172.21.15.195:0/1334238275 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b440f62c0 0x7f9b440f66e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f9b34009fb0 tx=0x7f9b34009b30).stop 2023-10-12T22:37:10.981 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.980+0000 7f9b49600700 1 -- 172.21.15.195:0/1334238275 shutdown_connections 2023-10-12T22:37:10.981 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.980+0000 7f9b49600700 1 --2- 172.21.15.195:0/1334238275 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b440f62c0 0x7f9b440f66e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:10.981 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.980+0000 7f9b49600700 1 -- 172.21.15.195:0/1334238275 >> 172.21.15.195:0/1334238275 conn(0x7f9b440f1a20 msgr2=0x7f9b440f3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:10.981 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.980+0000 7f9b49600700 1 -- 172.21.15.195:0/1334238275 shutdown_connections 2023-10-12T22:37:10.981 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.980+0000 7f9b49600700 1 -- 172.21.15.195:0/1334238275 wait complete. 2023-10-12T22:37:10.983 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.980+0000 7f9b49600700 1 Processor -- start 2023-10-12T22:37:10.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.980+0000 7f9b49600700 1 -- start start 2023-10-12T22:37:10.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.981+0000 7f9b49600700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b440f62c0 0x7f9b44106580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:10.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.981+0000 7f9b49600700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44106a70 con 0x7f9b440f62c0 2023-10-12T22:37:10.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.981+0000 7f9b42ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b440f62c0 0x7f9b44106580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:10.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.981+0000 7f9b42ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b440f62c0 0x7f9b44106580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:49468/0 (socket says 172.21.15.195:49468) 2023-10-12T22:37:10.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.981+0000 7f9b42ffd700 1 -- 172.21.15.195:0/2860964723 learned_addr learned my addr 172.21.15.195:0/2860964723 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:10.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.981+0000 7f9b42ffd700 1 -- 172.21.15.195:0/2860964723 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b34005040 con 0x7f9b440f62c0 2023-10-12T22:37:10.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.981+0000 7f9b42ffd700 1 --2- 172.21.15.195:0/2860964723 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b440f62c0 0x7f9b44106580 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f9b34014000 tx=0x7f9b3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:10.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.981+0000 7f9b2b7fe700 1 -- 172.21.15.195:0/2860964723 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b340095f0 con 0x7f9b440f62c0 2023-10-12T22:37:10.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.982+0000 7f9b49600700 1 -- 172.21.15.195:0/2860964723 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b44106c70 con 0x7f9b440f62c0 2023-10-12T22:37:10.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.982+0000 7f9b2b7fe700 1 -- 172.21.15.195:0/2860964723 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b34007e70 con 0x7f9b440f62c0 2023-10-12T22:37:10.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.982+0000 7f9b2b7fe700 1 -- 172.21.15.195:0/2860964723 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b34014ea0 con 0x7f9b440f62c0 2023-10-12T22:37:10.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.982+0000 7f9b49600700 1 -- 172.21.15.195:0/2860964723 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b44107100 con 0x7f9b440f62c0 2023-10-12T22:37:10.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.982+0000 7f9b2b7fe700 1 -- 172.21.15.195:0/2860964723 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b34025070 con 0x7f9b440f62c0 2023-10-12T22:37:10.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.982+0000 7f9b2b7fe700 1 --2- 172.21.15.195:0/2860964723 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9b2c02ff60 0x7f9b2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:10.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.982+0000 7f9b2b7fe700 1 -- 172.21.15.195:0/2860964723 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b34040960 con 0x7f9b440f62c0 2023-10-12T22:37:10.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.983+0000 7f9b49600700 1 -- 172.21.15.195:0/2860964723 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b44045ad0 con 0x7f9b440f62c0 2023-10-12T22:37:10.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.983+0000 7f9b427fc700 1 --2- 172.21.15.195:0/2860964723 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9b2c02ff60 0x7f9b2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:10.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.983+0000 7f9b427fc700 1 --2- 172.21.15.195:0/2860964723 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9b2c02ff60 0x7f9b2c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9b380096a0 tx=0x7f9b38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:10.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:10.988+0000 7f9b2b7fe700 1 -- 172.21.15.195:0/2860964723 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b34026350 con 0x7f9b440f62c0 2023-10-12T22:37:11.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:11.194+0000 7f9b49600700 1 -- 172.21.15.195:0/2860964723 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b44107d80 con 0x7f9b440f62c0 2023-10-12T22:37:11.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:11.195+0000 7f9b2b7fe700 1 -- 172.21.15.195:0/2860964723 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9b34012330 con 0x7f9b440f62c0 2023-10-12T22:37:11.195 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:11.196 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:11.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:11.196+0000 7f9b49600700 1 -- 172.21.15.195:0/2860964723 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9b2c02ff60 msgr2=0x7f9b2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:11.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:11.197+0000 7f9b49600700 1 --2- 172.21.15.195:0/2860964723 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9b2c02ff60 0x7f9b2c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9b380096a0 tx=0x7f9b38006b40).stop 2023-10-12T22:37:11.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:11.197+0000 7f9b49600700 1 -- 172.21.15.195:0/2860964723 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b440f62c0 msgr2=0x7f9b44106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:11.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:11.197+0000 7f9b49600700 1 --2- 172.21.15.195:0/2860964723 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b440f62c0 0x7f9b44106580 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f9b34014000 tx=0x7f9b3400b040).stop 2023-10-12T22:37:11.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:11.197+0000 7f9b49600700 1 -- 172.21.15.195:0/2860964723 shutdown_connections 2023-10-12T22:37:11.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:11.197+0000 7f9b49600700 1 --2- 172.21.15.195:0/2860964723 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9b2c02ff60 0x7f9b2c032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:11.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:11.197+0000 7f9b49600700 1 --2- 172.21.15.195:0/2860964723 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b440f62c0 0x7f9b44106580 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:11.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:11.197+0000 7f9b49600700 1 -- 172.21.15.195:0/2860964723 >> 172.21.15.195:0/2860964723 conn(0x7f9b440f1a20 msgr2=0x7f9b440f2580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:11.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:11.197+0000 7f9b49600700 1 -- 172.21.15.195:0/2860964723 shutdown_connections 2023-10-12T22:37:11.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:11.197+0000 7f9b49600700 1 -- 172.21.15.195:0/2860964723 wait complete. 2023-10-12T22:37:11.199 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:11.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:11.195398+0000 mon.smithi131 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.195:0/2860964723' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:12.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:12.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:12.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:10.850844+0000 mgr.smithi131.efktma (mgr.14162) 179 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:13.796 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.794+0000 7f9c324eb700 1 -- 172.21.15.195:0/410715986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c0f6510 msgr2=0x7f9c2c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:13.796 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.794+0000 7f9c324eb700 1 --2- 172.21.15.195:0/410715986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c0f6510 0x7f9c2c0f6930 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f9c1c009fb0 tx=0x7f9c1c009b30).stop 2023-10-12T22:37:13.797 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.794+0000 7f9c324eb700 1 -- 172.21.15.195:0/410715986 shutdown_connections 2023-10-12T22:37:13.797 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.794+0000 7f9c324eb700 1 --2- 172.21.15.195:0/410715986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c0f6510 0x7f9c2c0f6930 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:13.797 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.794+0000 7f9c324eb700 1 -- 172.21.15.195:0/410715986 >> 172.21.15.195:0/410715986 conn(0x7f9c2c0f1c70 msgr2=0x7f9c2c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:13.797 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.794+0000 7f9c324eb700 1 -- 172.21.15.195:0/410715986 shutdown_connections 2023-10-12T22:37:13.797 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.794+0000 7f9c324eb700 1 -- 172.21.15.195:0/410715986 wait complete. 2023-10-12T22:37:13.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.795+0000 7f9c324eb700 1 Processor -- start 2023-10-12T22:37:13.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.795+0000 7f9c324eb700 1 -- start start 2023-10-12T22:37:13.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.795+0000 7f9c324eb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c0f6510 0x7f9c2c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:13.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.795+0000 7f9c324eb700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c106c20 con 0x7f9c2c0f6510 2023-10-12T22:37:13.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.795+0000 7f9c2bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c0f6510 0x7f9c2c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:13.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.795+0000 7f9c2bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c0f6510 0x7f9c2c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:49478/0 (socket says 172.21.15.195:49478) 2023-10-12T22:37:13.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.795+0000 7f9c2bfff700 1 -- 172.21.15.195:0/3528194316 learned_addr learned my addr 172.21.15.195:0/3528194316 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:13.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.795+0000 7f9c2bfff700 1 -- 172.21.15.195:0/3528194316 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c005040 con 0x7f9c2c0f6510 2023-10-12T22:37:13.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.796+0000 7f9c2bfff700 1 --2- 172.21.15.195:0/3528194316 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c0f6510 0x7f9c2c106730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f9c1c013510 tx=0x7f9c1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:13.799 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.796+0000 7f9c28ff9700 1 -- 172.21.15.195:0/3528194316 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c1c009320 con 0x7f9c2c0f6510 2023-10-12T22:37:13.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.796+0000 7f9c28ff9700 1 -- 172.21.15.195:0/3528194316 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c1c013f00 con 0x7f9c2c0f6510 2023-10-12T22:37:13.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.796+0000 7f9c324eb700 1 -- 172.21.15.195:0/3528194316 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2c106e20 con 0x7f9c2c0f6510 2023-10-12T22:37:13.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.796+0000 7f9c28ff9700 1 -- 172.21.15.195:0/3528194316 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c1c014d90 con 0x7f9c2c0f6510 2023-10-12T22:37:13.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.796+0000 7f9c324eb700 1 -- 172.21.15.195:0/3528194316 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2c1072b0 con 0x7f9c2c0f6510 2023-10-12T22:37:13.801 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.797+0000 7f9c28ff9700 1 -- 172.21.15.195:0/3528194316 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c1c0095f0 con 0x7f9c2c0f6510 2023-10-12T22:37:13.801 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.797+0000 7f9c324eb700 1 -- 172.21.15.195:0/3528194316 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2c045ad0 con 0x7f9c2c0f6510 2023-10-12T22:37:13.802 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.797+0000 7f9c28ff9700 1 --2- 172.21.15.195:0/3528194316 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9c1402ff60 0x7f9c14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:13.802 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.797+0000 7f9c28ff9700 1 -- 172.21.15.195:0/3528194316 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c1c040fe0 con 0x7f9c2c0f6510 2023-10-12T22:37:13.802 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.797+0000 7f9c2b7fe700 1 --2- 172.21.15.195:0/3528194316 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9c1402ff60 0x7f9c14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:13.802 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.797+0000 7f9c2b7fe700 1 --2- 172.21.15.195:0/3528194316 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9c1402ff60 0x7f9c14032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f9c200096a0 tx=0x7f9c20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:13.803 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:13.802+0000 7f9c28ff9700 1 -- 172.21.15.195:0/3528194316 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c1c014420 con 0x7f9c2c0f6510 2023-10-12T22:37:14.008 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:14.008+0000 7f9c324eb700 1 -- 172.21.15.195:0/3528194316 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c2c0fbf60 con 0x7f9c2c0f6510 2023-10-12T22:37:14.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:14.009+0000 7f9c28ff9700 1 -- 172.21.15.195:0/3528194316 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9c1c010230 con 0x7f9c2c0f6510 2023-10-12T22:37:14.009 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:14.010 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:14.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:14.010+0000 7f9c324eb700 1 -- 172.21.15.195:0/3528194316 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9c1402ff60 msgr2=0x7f9c14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:14.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:14.010+0000 7f9c324eb700 1 --2- 172.21.15.195:0/3528194316 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9c1402ff60 0x7f9c14032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f9c200096a0 tx=0x7f9c20006b40).stop 2023-10-12T22:37:14.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:14.010+0000 7f9c324eb700 1 -- 172.21.15.195:0/3528194316 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c0f6510 msgr2=0x7f9c2c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:14.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:14.010+0000 7f9c324eb700 1 --2- 172.21.15.195:0/3528194316 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c0f6510 0x7f9c2c106730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f9c1c013510 tx=0x7f9c1c00b040).stop 2023-10-12T22:37:14.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:14.010+0000 7f9c324eb700 1 -- 172.21.15.195:0/3528194316 shutdown_connections 2023-10-12T22:37:14.011 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:14.010+0000 7f9c324eb700 1 --2- 172.21.15.195:0/3528194316 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9c1402ff60 0x7f9c14032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:14.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:14.010+0000 7f9c324eb700 1 --2- 172.21.15.195:0/3528194316 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c0f6510 0x7f9c2c106730 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:14.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:14.010+0000 7f9c324eb700 1 -- 172.21.15.195:0/3528194316 >> 172.21.15.195:0/3528194316 conn(0x7f9c2c0f1c70 msgr2=0x7f9c2c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:14.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:14.010+0000 7f9c324eb700 1 -- 172.21.15.195:0/3528194316 shutdown_connections 2023-10-12T22:37:14.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:14.010+0000 7f9c324eb700 1 -- 172.21.15.195:0/3528194316 wait complete. 2023-10-12T22:37:14.013 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:14.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:14 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:12.851258+0000 mgr.smithi131.efktma (mgr.14162) 180 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:14.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:14 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:14.009060+0000 mon.smithi131 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.195:0/3528194316' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:15.402 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:15.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:16.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.586+0000 7f41bbfff700 1 -- 172.21.15.195:0/133698826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41bc0f62a0 msgr2=0x7f41bc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:16.586 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.586+0000 7f41bbfff700 1 --2- 172.21.15.195:0/133698826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41bc0f62a0 0x7f41bc0f66c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f41b0009fb0 tx=0x7f41b0009b30).stop 2023-10-12T22:37:16.587 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.586+0000 7f41bbfff700 1 -- 172.21.15.195:0/133698826 shutdown_connections 2023-10-12T22:37:16.587 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.586+0000 7f41bbfff700 1 --2- 172.21.15.195:0/133698826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41bc0f62a0 0x7f41bc0f66c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:16.587 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.586+0000 7f41bbfff700 1 -- 172.21.15.195:0/133698826 >> 172.21.15.195:0/133698826 conn(0x7f41bc0f1aa0 msgr2=0x7f41bc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:16.587 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.586+0000 7f41bbfff700 1 -- 172.21.15.195:0/133698826 shutdown_connections 2023-10-12T22:37:16.587 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.586+0000 7f41bbfff700 1 -- 172.21.15.195:0/133698826 wait complete. 2023-10-12T22:37:16.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.587+0000 7f41bbfff700 1 Processor -- start 2023-10-12T22:37:16.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.587+0000 7f41bbfff700 1 -- start start 2023-10-12T22:37:16.590 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.587+0000 7f41bbfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41bc0f62a0 0x7f41bc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:16.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.587+0000 7f41bbfff700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc1069a0 con 0x7f41bc0f62a0 2023-10-12T22:37:16.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.587+0000 7f41baffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41bc0f62a0 0x7f41bc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:16.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.587+0000 7f41baffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41bc0f62a0 0x7f41bc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:49508/0 (socket says 172.21.15.195:49508) 2023-10-12T22:37:16.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.587+0000 7f41baffd700 1 -- 172.21.15.195:0/3642947861 learned_addr learned my addr 172.21.15.195:0/3642947861 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:16.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.587+0000 7f41baffd700 1 -- 172.21.15.195:0/3642947861 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b0005040 con 0x7f41bc0f62a0 2023-10-12T22:37:16.591 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.587+0000 7f41baffd700 1 --2- 172.21.15.195:0/3642947861 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41bc0f62a0 0x7f41bc1064b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f41b0013510 tx=0x7f41b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:16.592 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.587+0000 7f41a37fe700 1 -- 172.21.15.195:0/3642947861 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41b0009320 con 0x7f41bc0f62a0 2023-10-12T22:37:16.592 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.588+0000 7f41bbfff700 1 -- 172.21.15.195:0/3642947861 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41bc106ba0 con 0x7f41bc0f62a0 2023-10-12T22:37:16.592 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.588+0000 7f41a37fe700 1 -- 172.21.15.195:0/3642947861 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41b0013f00 con 0x7f41bc0f62a0 2023-10-12T22:37:16.592 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.588+0000 7f41a37fe700 1 -- 172.21.15.195:0/3642947861 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41b0014dc0 con 0x7f41bc0f62a0 2023-10-12T22:37:16.592 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.588+0000 7f41bbfff700 1 -- 172.21.15.195:0/3642947861 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41bc107030 con 0x7f41bc0f62a0 2023-10-12T22:37:16.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.588+0000 7f41a37fe700 1 -- 172.21.15.195:0/3642947861 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41b0014420 con 0x7f41bc0f62a0 2023-10-12T22:37:16.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.588+0000 7f41bbfff700 1 -- 172.21.15.195:0/3642947861 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41bc045ad0 con 0x7f41bc0f62a0 2023-10-12T22:37:16.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.589+0000 7f41a37fe700 1 --2- 172.21.15.195:0/3642947861 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f41a402ff60 0x7f41a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:16.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.589+0000 7f41a37fe700 1 -- 172.21.15.195:0/3642947861 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f41b0041020 con 0x7f41bc0f62a0 2023-10-12T22:37:16.593 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.589+0000 7f41ba7fc700 1 --2- 172.21.15.195:0/3642947861 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f41a402ff60 0x7f41a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:16.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.589+0000 7f41ba7fc700 1 --2- 172.21.15.195:0/3642947861 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f41a402ff60 0x7f41a4032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f41ac0096a0 tx=0x7f41ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:16.594 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.593+0000 7f41a37fe700 1 -- 172.21.15.195:0/3642947861 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41b00095f0 con 0x7f41bc0f62a0 2023-10-12T22:37:16.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:16 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:14.851530+0000 mgr.smithi131.efktma (mgr.14162) 181 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:16.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.797+0000 7f41bbfff700 1 -- 172.21.15.195:0/3642947861 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41bc107ee0 con 0x7f41bc0f62a0 2023-10-12T22:37:16.798 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.798+0000 7f41a37fe700 1 -- 172.21.15.195:0/3642947861 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f41b001cb10 con 0x7f41bc0f62a0 2023-10-12T22:37:16.799 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:16.799 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:16.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.799+0000 7f41bbfff700 1 -- 172.21.15.195:0/3642947861 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f41a402ff60 msgr2=0x7f41a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:16.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.799+0000 7f41bbfff700 1 --2- 172.21.15.195:0/3642947861 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f41a402ff60 0x7f41a4032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f41ac0096a0 tx=0x7f41ac006b40).stop 2023-10-12T22:37:16.800 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.799+0000 7f41bbfff700 1 -- 172.21.15.195:0/3642947861 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41bc0f62a0 msgr2=0x7f41bc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:16.801 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.799+0000 7f41bbfff700 1 --2- 172.21.15.195:0/3642947861 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41bc0f62a0 0x7f41bc1064b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f41b0013510 tx=0x7f41b000b040).stop 2023-10-12T22:37:16.801 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.799+0000 7f41bbfff700 1 -- 172.21.15.195:0/3642947861 shutdown_connections 2023-10-12T22:37:16.801 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.799+0000 7f41bbfff700 1 --2- 172.21.15.195:0/3642947861 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f41a402ff60 0x7f41a4032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:16.801 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.799+0000 7f41bbfff700 1 --2- 172.21.15.195:0/3642947861 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f41bc0f62a0 0x7f41bc1064b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:16.801 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.799+0000 7f41bbfff700 1 -- 172.21.15.195:0/3642947861 >> 172.21.15.195:0/3642947861 conn(0x7f41bc0f1aa0 msgr2=0x7f41bc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:16.802 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.800+0000 7f41bbfff700 1 -- 172.21.15.195:0/3642947861 shutdown_connections 2023-10-12T22:37:16.802 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:16.800+0000 7f41bbfff700 1 -- 172.21.15.195:0/3642947861 wait complete. 2023-10-12T22:37:16.802 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:17.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:17 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:16.798435+0000 mon.smithi131 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.195:0/3642947861' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:18.192 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:18.192 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:18.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:18 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:16.851781+0000 mgr.smithi131.efktma (mgr.14162) 182 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:19.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.349+0000 7f5b98015700 1 -- 172.21.15.195:0/1799758807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b900f5410 msgr2=0x7f5b900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:19.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.349+0000 7f5b98015700 1 --2- 172.21.15.195:0/1799758807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b900f5410 0x7f5b900f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f5b80009fb0 tx=0x7f5b80009b30).stop 2023-10-12T22:37:19.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.349+0000 7f5b98015700 1 -- 172.21.15.195:0/1799758807 shutdown_connections 2023-10-12T22:37:19.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.349+0000 7f5b98015700 1 --2- 172.21.15.195:0/1799758807 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b900f5410 0x7f5b900f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:19.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.349+0000 7f5b98015700 1 -- 172.21.15.195:0/1799758807 >> 172.21.15.195:0/1799758807 conn(0x7f5b900f0b70 msgr2=0x7f5b900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:19.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.349+0000 7f5b98015700 1 -- 172.21.15.195:0/1799758807 shutdown_connections 2023-10-12T22:37:19.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.349+0000 7f5b98015700 1 -- 172.21.15.195:0/1799758807 wait complete. 2023-10-12T22:37:19.354 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.349+0000 7f5b98015700 1 Processor -- start 2023-10-12T22:37:19.354 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.349+0000 7f5b98015700 1 -- start start 2023-10-12T22:37:19.354 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.349+0000 7f5b98015700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b900f5410 0x7f5b900fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:19.354 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.349+0000 7f5b98015700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b900fb820 con 0x7f5b900f5410 2023-10-12T22:37:19.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.350+0000 7f5b95db1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b900f5410 0x7f5b900fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:19.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.350+0000 7f5b95db1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b900f5410 0x7f5b900fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:42398/0 (socket says 172.21.15.195:42398) 2023-10-12T22:37:19.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.350+0000 7f5b95db1700 1 -- 172.21.15.195:0/558060519 learned_addr learned my addr 172.21.15.195:0/558060519 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:19.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.350+0000 7f5b95db1700 1 -- 172.21.15.195:0/558060519 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b80005040 con 0x7f5b900f5410 2023-10-12T22:37:19.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.350+0000 7f5b95db1700 1 --2- 172.21.15.195:0/558060519 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b900f5410 0x7f5b900fd3a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f5b80014000 tx=0x7f5b8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:19.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.350+0000 7f5b867fc700 1 -- 172.21.15.195:0/558060519 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b800095f0 con 0x7f5b900f5410 2023-10-12T22:37:19.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.350+0000 7f5b98015700 1 -- 172.21.15.195:0/558060519 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b900fb9c0 con 0x7f5b900f5410 2023-10-12T22:37:19.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.350+0000 7f5b867fc700 1 -- 172.21.15.195:0/558060519 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b80007e70 con 0x7f5b900f5410 2023-10-12T22:37:19.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.350+0000 7f5b867fc700 1 -- 172.21.15.195:0/558060519 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b80014ea0 con 0x7f5b900f5410 2023-10-12T22:37:19.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.350+0000 7f5b98015700 1 -- 172.21.15.195:0/558060519 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b900fbe50 con 0x7f5b900f5410 2023-10-12T22:37:19.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.351+0000 7f5b867fc700 1 -- 172.21.15.195:0/558060519 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b800079c0 con 0x7f5b900f5410 2023-10-12T22:37:19.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.351+0000 7f5b98015700 1 -- 172.21.15.195:0/558060519 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b90059100 con 0x7f5b900f5410 2023-10-12T22:37:19.358 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.351+0000 7f5b867fc700 1 --2- 172.21.15.195:0/558060519 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5b7c02ff60 0x7f5b7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:19.358 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.351+0000 7f5b867fc700 1 -- 172.21.15.195:0/558060519 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b80040860 con 0x7f5b900f5410 2023-10-12T22:37:19.358 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.351+0000 7f5b955b0700 1 --2- 172.21.15.195:0/558060519 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5b7c02ff60 0x7f5b7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:19.359 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.352+0000 7f5b955b0700 1 --2- 172.21.15.195:0/558060519 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5b7c02ff60 0x7f5b7c032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f5b8c0096a0 tx=0x7f5b8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:19.359 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.356+0000 7f5b867fc700 1 -- 172.21.15.195:0/558060519 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b8000f950 con 0x7f5b900f5410 2023-10-12T22:37:19.563 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.562+0000 7f5b98015700 1 -- 172.21.15.195:0/558060519 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b900fae60 con 0x7f5b900f5410 2023-10-12T22:37:19.564 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.563+0000 7f5b867fc700 1 -- 172.21.15.195:0/558060519 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5b80010230 con 0x7f5b900f5410 2023-10-12T22:37:19.564 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:19.564 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:19.565 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.565+0000 7f5b98015700 1 -- 172.21.15.195:0/558060519 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5b7c02ff60 msgr2=0x7f5b7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:19.566 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.565+0000 7f5b98015700 1 --2- 172.21.15.195:0/558060519 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5b7c02ff60 0x7f5b7c032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f5b8c0096a0 tx=0x7f5b8c006b40).stop 2023-10-12T22:37:19.566 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.565+0000 7f5b98015700 1 -- 172.21.15.195:0/558060519 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b900f5410 msgr2=0x7f5b900fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:19.566 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.565+0000 7f5b98015700 1 --2- 172.21.15.195:0/558060519 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b900f5410 0x7f5b900fd3a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f5b80014000 tx=0x7f5b8000b040).stop 2023-10-12T22:37:19.567 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.565+0000 7f5b98015700 1 -- 172.21.15.195:0/558060519 shutdown_connections 2023-10-12T22:37:19.567 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.565+0000 7f5b98015700 1 --2- 172.21.15.195:0/558060519 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5b7c02ff60 0x7f5b7c032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:19.567 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.565+0000 7f5b98015700 1 --2- 172.21.15.195:0/558060519 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b900f5410 0x7f5b900fd3a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:19.567 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.565+0000 7f5b98015700 1 -- 172.21.15.195:0/558060519 >> 172.21.15.195:0/558060519 conn(0x7f5b900f0b70 msgr2=0x7f5b9005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:19.568 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.565+0000 7f5b98015700 1 -- 172.21.15.195:0/558060519 shutdown_connections 2023-10-12T22:37:19.568 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:19.565+0000 7f5b98015700 1 -- 172.21.15.195:0/558060519 wait complete. 2023-10-12T22:37:19.568 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:20.742 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:20 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:18.851960+0000 mgr.smithi131.efktma (mgr.14162) 183 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:20.742 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:20 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:19.563691+0000 mon.smithi131 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.195:0/558060519' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:20.938 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:20.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:22.083 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.081+0000 7f5e4ed72700 1 -- 172.21.15.195:0/3553212666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e480f5410 msgr2=0x7f5e480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:22.083 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.081+0000 7f5e4ed72700 1 --2- 172.21.15.195:0/3553212666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e480f5410 0x7f5e480f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f5e3c009fb0 tx=0x7f5e3c009b30).stop 2023-10-12T22:37:22.083 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.081+0000 7f5e4ed72700 1 -- 172.21.15.195:0/3553212666 shutdown_connections 2023-10-12T22:37:22.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.081+0000 7f5e4ed72700 1 --2- 172.21.15.195:0/3553212666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e480f5410 0x7f5e480f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:22.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.081+0000 7f5e4ed72700 1 -- 172.21.15.195:0/3553212666 >> 172.21.15.195:0/3553212666 conn(0x7f5e480f0b70 msgr2=0x7f5e480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:22.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.081+0000 7f5e4ed72700 1 -- 172.21.15.195:0/3553212666 shutdown_connections 2023-10-12T22:37:22.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.081+0000 7f5e4ed72700 1 -- 172.21.15.195:0/3553212666 wait complete. 2023-10-12T22:37:22.084 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.081+0000 7f5e4ed72700 1 Processor -- start 2023-10-12T22:37:22.085 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.082+0000 7f5e4ed72700 1 -- start start 2023-10-12T22:37:22.085 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.082+0000 7f5e4ed72700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e480f5410 0x7f5e48104520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:22.085 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.082+0000 7f5e4ed72700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e48104a10 con 0x7f5e480f5410 2023-10-12T22:37:22.085 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.082+0000 7f5e4cb0e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e480f5410 0x7f5e48104520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:22.085 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.082+0000 7f5e4cb0e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e480f5410 0x7f5e48104520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:42424/0 (socket says 172.21.15.195:42424) 2023-10-12T22:37:22.085 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.082+0000 7f5e4cb0e700 1 -- 172.21.15.195:0/18778549 learned_addr learned my addr 172.21.15.195:0/18778549 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:22.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.082+0000 7f5e4cb0e700 1 -- 172.21.15.195:0/18778549 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c005040 con 0x7f5e480f5410 2023-10-12T22:37:22.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.082+0000 7f5e4cb0e700 1 --2- 172.21.15.195:0/18778549 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e480f5410 0x7f5e48104520 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f5e3c013510 tx=0x7f5e3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:22.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.083+0000 7f5e457fa700 1 -- 172.21.15.195:0/18778549 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e3c009320 con 0x7f5e480f5410 2023-10-12T22:37:22.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.083+0000 7f5e457fa700 1 -- 172.21.15.195:0/18778549 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e3c013f00 con 0x7f5e480f5410 2023-10-12T22:37:22.086 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.083+0000 7f5e4ed72700 1 -- 172.21.15.195:0/18778549 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e48104c10 con 0x7f5e480f5410 2023-10-12T22:37:22.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.083+0000 7f5e457fa700 1 -- 172.21.15.195:0/18778549 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e3c014d90 con 0x7f5e480f5410 2023-10-12T22:37:22.087 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.083+0000 7f5e4ed72700 1 -- 172.21.15.195:0/18778549 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e481050a0 con 0x7f5e480f5410 2023-10-12T22:37:22.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.084+0000 7f5e457fa700 1 -- 172.21.15.195:0/18778549 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e3c0263a0 con 0x7f5e480f5410 2023-10-12T22:37:22.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.084+0000 7f5e4ed72700 1 -- 172.21.15.195:0/18778549 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e48045ad0 con 0x7f5e480f5410 2023-10-12T22:37:22.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.084+0000 7f5e457fa700 1 --2- 172.21.15.195:0/18778549 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5e380321b0 0x7f5e38034670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:22.088 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.084+0000 7f5e457fa700 1 -- 172.21.15.195:0/18778549 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e3c0412c0 con 0x7f5e480f5410 2023-10-12T22:37:22.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.084+0000 7f5e47fff700 1 --2- 172.21.15.195:0/18778549 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5e380321b0 0x7f5e38034670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:22.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.084+0000 7f5e47fff700 1 --2- 172.21.15.195:0/18778549 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5e380321b0 0x7f5e38034670 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5e340096a0 tx=0x7f5e34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:22.089 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.088+0000 7f5e457fa700 1 -- 172.21.15.195:0/18778549 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e3c0126f0 con 0x7f5e480f5410 2023-10-12T22:37:22.297 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.297+0000 7f5e4ed72700 1 -- 172.21.15.195:0/18778549 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e480f9d80 con 0x7f5e480f5410 2023-10-12T22:37:22.298 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.298+0000 7f5e457fa700 1 -- 172.21.15.195:0/18778549 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5e3c012330 con 0x7f5e480f5410 2023-10-12T22:37:22.298 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:22.298 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:22.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.299+0000 7f5e4ed72700 1 -- 172.21.15.195:0/18778549 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5e380321b0 msgr2=0x7f5e38034670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:22.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.299+0000 7f5e4ed72700 1 --2- 172.21.15.195:0/18778549 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5e380321b0 0x7f5e38034670 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5e340096a0 tx=0x7f5e34006b40).stop 2023-10-12T22:37:22.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.299+0000 7f5e4ed72700 1 -- 172.21.15.195:0/18778549 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e480f5410 msgr2=0x7f5e48104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:22.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.299+0000 7f5e4ed72700 1 --2- 172.21.15.195:0/18778549 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e480f5410 0x7f5e48104520 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f5e3c013510 tx=0x7f5e3c00b040).stop 2023-10-12T22:37:22.300 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.299+0000 7f5e4ed72700 1 -- 172.21.15.195:0/18778549 shutdown_connections 2023-10-12T22:37:22.301 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.299+0000 7f5e4ed72700 1 --2- 172.21.15.195:0/18778549 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f5e380321b0 0x7f5e38034670 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:22.301 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.299+0000 7f5e4ed72700 1 --2- 172.21.15.195:0/18778549 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e480f5410 0x7f5e48104520 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:22.301 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.299+0000 7f5e4ed72700 1 -- 172.21.15.195:0/18778549 >> 172.21.15.195:0/18778549 conn(0x7f5e480f0b70 msgr2=0x7f5e480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:22.301 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.299+0000 7f5e4ed72700 1 -- 172.21.15.195:0/18778549 shutdown_connections 2023-10-12T22:37:22.301 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:22.300+0000 7f5e4ed72700 1 -- 172.21.15.195:0/18778549 wait complete. 2023-10-12T22:37:22.302 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:22.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:20.852274+0000 mgr.smithi131.efktma (mgr.14162) 184 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:22.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:22.297999+0000 mon.smithi131 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.195:0/18778549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:23.669 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:23.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:23.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:22.937694+0000 mon.smithi131 (mon.0) 254 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/mirror_snapshot_schedule"}]: dispatch 2023-10-12T22:37:23.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:22.942685+0000 mon.smithi131 (mon.0) 255 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/trash_purge_schedule"}]: dispatch 2023-10-12T22:37:24.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:22.852548+0000 mgr.smithi131.efktma (mgr.14162) 185 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:24.831 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.829+0000 7f0db044f700 1 -- 172.21.15.195:0/3774171650 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0da80f5410 msgr2=0x7f0da80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:24.832 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.829+0000 7f0db044f700 1 --2- 172.21.15.195:0/3774171650 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0da80f5410 0x7f0da80f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f0d9c009fb0 tx=0x7f0d9c009b30).stop 2023-10-12T22:37:24.832 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.829+0000 7f0db044f700 1 -- 172.21.15.195:0/3774171650 shutdown_connections 2023-10-12T22:37:24.832 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.829+0000 7f0db044f700 1 --2- 172.21.15.195:0/3774171650 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0da80f5410 0x7f0da80f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:24.832 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.829+0000 7f0db044f700 1 -- 172.21.15.195:0/3774171650 >> 172.21.15.195:0/3774171650 conn(0x7f0da80f0b70 msgr2=0x7f0da80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:24.833 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.829+0000 7f0db044f700 1 -- 172.21.15.195:0/3774171650 shutdown_connections 2023-10-12T22:37:24.833 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.829+0000 7f0db044f700 1 -- 172.21.15.195:0/3774171650 wait complete. 2023-10-12T22:37:24.833 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.830+0000 7f0db044f700 1 Processor -- start 2023-10-12T22:37:24.833 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.830+0000 7f0db044f700 1 -- start start 2023-10-12T22:37:24.833 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.830+0000 7f0db044f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0da80f5410 0x7f0da8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:24.834 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.830+0000 7f0db044f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da8105b20 con 0x7f0da80f5410 2023-10-12T22:37:24.834 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.830+0000 7f0dae1eb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0da80f5410 0x7f0da8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:24.834 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.830+0000 7f0dae1eb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0da80f5410 0x7f0da8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:42436/0 (socket says 172.21.15.195:42436) 2023-10-12T22:37:24.834 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.830+0000 7f0dae1eb700 1 -- 172.21.15.195:0/753324843 learned_addr learned my addr 172.21.15.195:0/753324843 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:24.834 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.831+0000 7f0dae1eb700 1 -- 172.21.15.195:0/753324843 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c005040 con 0x7f0da80f5410 2023-10-12T22:37:24.836 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.831+0000 7f0dae1eb700 1 --2- 172.21.15.195:0/753324843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0da80f5410 0x7f0da8105630 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f0d9c013510 tx=0x7f0d9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:24.836 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.831+0000 7f0d9affd700 1 -- 172.21.15.195:0/753324843 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9c009320 con 0x7f0da80f5410 2023-10-12T22:37:24.836 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.831+0000 7f0d9affd700 1 -- 172.21.15.195:0/753324843 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d9c013f00 con 0x7f0da80f5410 2023-10-12T22:37:24.836 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.831+0000 7f0db044f700 1 -- 172.21.15.195:0/753324843 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da8105d20 con 0x7f0da80f5410 2023-10-12T22:37:24.837 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.831+0000 7f0d9affd700 1 -- 172.21.15.195:0/753324843 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9c014d90 con 0x7f0da80f5410 2023-10-12T22:37:24.837 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.831+0000 7f0db044f700 1 -- 172.21.15.195:0/753324843 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da81061b0 con 0x7f0da80f5410 2023-10-12T22:37:24.837 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.832+0000 7f0d9affd700 1 -- 172.21.15.195:0/753324843 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d9c00f410 con 0x7f0da80f5410 2023-10-12T22:37:24.837 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.832+0000 7f0d9affd700 1 --2- 172.21.15.195:0/753324843 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0d9402ff60 0x7f0d94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:24.837 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.832+0000 7f0db044f700 1 -- 172.21.15.195:0/753324843 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da8045ad0 con 0x7f0da80f5410 2023-10-12T22:37:24.838 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.832+0000 7f0d9affd700 1 -- 172.21.15.195:0/753324843 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d9c043140 con 0x7f0da80f5410 2023-10-12T22:37:24.838 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.832+0000 7f0dad9ea700 1 --2- 172.21.15.195:0/753324843 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0d9402ff60 0x7f0d94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:24.838 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.832+0000 7f0dad9ea700 1 --2- 172.21.15.195:0/753324843 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0d9402ff60 0x7f0d94032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f0da40096a0 tx=0x7f0da4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:24.838 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:24.836+0000 7f0d9affd700 1 -- 172.21.15.195:0/753324843 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d9c0283a0 con 0x7f0da80f5410 2023-10-12T22:37:25.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:25.043+0000 7f0db044f700 1 -- 172.21.15.195:0/753324843 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0da80faf50 con 0x7f0da80f5410 2023-10-12T22:37:25.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:25.044+0000 7f0d9affd700 1 -- 172.21.15.195:0/753324843 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0d9c03d090 con 0x7f0da80f5410 2023-10-12T22:37:25.045 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:25.046 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:25.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:25.045+0000 7f0db044f700 1 -- 172.21.15.195:0/753324843 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0d9402ff60 msgr2=0x7f0d94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:25.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:25.045+0000 7f0db044f700 1 --2- 172.21.15.195:0/753324843 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0d9402ff60 0x7f0d94032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f0da40096a0 tx=0x7f0da4006b40).stop 2023-10-12T22:37:25.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:25.045+0000 7f0db044f700 1 -- 172.21.15.195:0/753324843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0da80f5410 msgr2=0x7f0da8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:25.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:25.045+0000 7f0db044f700 1 --2- 172.21.15.195:0/753324843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0da80f5410 0x7f0da8105630 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f0d9c013510 tx=0x7f0d9c00b040).stop 2023-10-12T22:37:25.047 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:25.045+0000 7f0db044f700 1 -- 172.21.15.195:0/753324843 shutdown_connections 2023-10-12T22:37:25.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:25.045+0000 7f0db044f700 1 --2- 172.21.15.195:0/753324843 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f0d9402ff60 0x7f0d94032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:25.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:25.045+0000 7f0db044f700 1 --2- 172.21.15.195:0/753324843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0da80f5410 0x7f0da8105630 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:25.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:25.045+0000 7f0db044f700 1 -- 172.21.15.195:0/753324843 >> 172.21.15.195:0/753324843 conn(0x7f0da80f0b70 msgr2=0x7f0da80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:25.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:25.046+0000 7f0db044f700 1 -- 172.21.15.195:0/753324843 shutdown_connections 2023-10-12T22:37:25.048 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:25.046+0000 7f0db044f700 1 -- 172.21.15.195:0/753324843 wait complete. 2023-10-12T22:37:25.049 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:26.202 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:25 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:24.852779+0000 mgr.smithi131.efktma (mgr.14162) 186 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:26.202 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:25 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:25.045275+0000 mon.smithi131 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.195:0/753324843' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:26.423 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:26.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:27.526 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.525+0000 7eff05370700 1 -- 172.21.15.195:0/1539727539 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff00060200 msgr2=0x7eff000f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:27.527 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.525+0000 7eff05370700 1 --2- 172.21.15.195:0/1539727539 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff00060200 0x7eff000f9170 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7efee8009fb0 tx=0x7efee8009b30).stop 2023-10-12T22:37:27.527 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.525+0000 7eff05370700 1 -- 172.21.15.195:0/1539727539 shutdown_connections 2023-10-12T22:37:27.527 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.525+0000 7eff05370700 1 --2- 172.21.15.195:0/1539727539 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff00060200 0x7eff000f9170 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:27.528 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.525+0000 7eff05370700 1 -- 172.21.15.195:0/1539727539 >> 172.21.15.195:0/1539727539 conn(0x7eff000f0b70 msgr2=0x7eff000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:27.528 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.525+0000 7eff05370700 1 -- 172.21.15.195:0/1539727539 shutdown_connections 2023-10-12T22:37:27.528 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.525+0000 7eff05370700 1 -- 172.21.15.195:0/1539727539 wait complete. 2023-10-12T22:37:27.530 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.526+0000 7eff05370700 1 Processor -- start 2023-10-12T22:37:27.530 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.526+0000 7eff05370700 1 -- start start 2023-10-12T22:37:27.530 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.526+0000 7eff05370700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff00060200 0x7eff000f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:27.530 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.526+0000 7eff05370700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff000f5070 con 0x7eff00060200 2023-10-12T22:37:27.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.526+0000 7efefeffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff00060200 0x7eff000f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:27.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.526+0000 7efefeffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff00060200 0x7eff000f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:42450/0 (socket says 172.21.15.195:42450) 2023-10-12T22:37:27.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.526+0000 7efefeffd700 1 -- 172.21.15.195:0/2001121099 learned_addr learned my addr 172.21.15.195:0/2001121099 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:27.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.527+0000 7efefeffd700 1 -- 172.21.15.195:0/2001121099 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee8005040 con 0x7eff00060200 2023-10-12T22:37:27.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.527+0000 7efefeffd700 1 --2- 172.21.15.195:0/2001121099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff00060200 0x7eff000f6bf0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7efee8013510 tx=0x7efee800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:27.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.527+0000 7efef77fe700 1 -- 172.21.15.195:0/2001121099 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efee8009320 con 0x7eff00060200 2023-10-12T22:37:27.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.527+0000 7efef77fe700 1 -- 172.21.15.195:0/2001121099 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efee8013f00 con 0x7eff00060200 2023-10-12T22:37:27.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.527+0000 7eff05370700 1 -- 172.21.15.195:0/2001121099 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff000f5210 con 0x7eff00060200 2023-10-12T22:37:27.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.527+0000 7efef77fe700 1 -- 172.21.15.195:0/2001121099 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efee8014dc0 con 0x7eff00060200 2023-10-12T22:37:27.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.527+0000 7eff05370700 1 -- 172.21.15.195:0/2001121099 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff000f56a0 con 0x7eff00060200 2023-10-12T22:37:27.533 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.528+0000 7efef77fe700 1 -- 172.21.15.195:0/2001121099 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efee801d3d0 con 0x7eff00060200 2023-10-12T22:37:27.533 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.528+0000 7eff05370700 1 -- 172.21.15.195:0/2001121099 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff00059100 con 0x7eff00060200 2023-10-12T22:37:27.533 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.528+0000 7efef77fe700 1 --2- 172.21.15.195:0/2001121099 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7efeec02ff60 0x7efeec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:27.533 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.528+0000 7efef77fe700 1 -- 172.21.15.195:0/2001121099 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efee80413a0 con 0x7eff00060200 2023-10-12T22:37:27.533 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.528+0000 7efefe7fc700 1 --2- 172.21.15.195:0/2001121099 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7efeec02ff60 0x7efeec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:27.534 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.528+0000 7efefe7fc700 1 --2- 172.21.15.195:0/2001121099 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7efeec02ff60 0x7efeec032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7efef00096a0 tx=0x7efef0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:27.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.534+0000 7efef77fe700 1 -- 172.21.15.195:0/2001121099 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efee801cb10 con 0x7eff00060200 2023-10-12T22:37:27.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.741+0000 7eff05370700 1 -- 172.21.15.195:0/2001121099 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff000fbf50 con 0x7eff00060200 2023-10-12T22:37:27.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.741+0000 7efef77fe700 1 -- 172.21.15.195:0/2001121099 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efee801cb10 con 0x7eff00060200 2023-10-12T22:37:27.743 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:27.743 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:27.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.743+0000 7eff05370700 1 -- 172.21.15.195:0/2001121099 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7efeec02ff60 msgr2=0x7efeec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:27.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.743+0000 7eff05370700 1 --2- 172.21.15.195:0/2001121099 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7efeec02ff60 0x7efeec032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7efef00096a0 tx=0x7efef0006b40).stop 2023-10-12T22:37:27.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.743+0000 7eff05370700 1 -- 172.21.15.195:0/2001121099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff00060200 msgr2=0x7eff000f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:27.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.743+0000 7eff05370700 1 --2- 172.21.15.195:0/2001121099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff00060200 0x7eff000f6bf0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7efee8013510 tx=0x7efee800b040).stop 2023-10-12T22:37:27.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.743+0000 7eff05370700 1 -- 172.21.15.195:0/2001121099 shutdown_connections 2023-10-12T22:37:27.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.743+0000 7eff05370700 1 --2- 172.21.15.195:0/2001121099 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7efeec02ff60 0x7efeec032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:27.745 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.743+0000 7eff05370700 1 --2- 172.21.15.195:0/2001121099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff00060200 0x7eff000f6bf0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:27.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.743+0000 7eff05370700 1 -- 172.21.15.195:0/2001121099 >> 172.21.15.195:0/2001121099 conn(0x7eff000f0b70 msgr2=0x7eff000f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:27.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.743+0000 7eff05370700 1 -- 172.21.15.195:0/2001121099 shutdown_connections 2023-10-12T22:37:27.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:27.743+0000 7eff05370700 1 -- 172.21.15.195:0/2001121099 wait complete. 2023-10-12T22:37:27.746 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:28.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:28 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:26.853028+0000 mgr.smithi131.efktma (mgr.14162) 187 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:28.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:28 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:27.585990+0000 mon.smithi131 (mon.0) 257 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' 2023-10-12T22:37:28.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:28 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:37:27.587369+0000 mgr.smithi131.efktma (mgr.14162) 188 : cephadm [INF] Deploying daemon node-exporter.smithi131 on smithi131 2023-10-12T22:37:28.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:28 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:27.742941+0000 mon.smithi131 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.195:0/2001121099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:29.113 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:29.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:30.301 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.299+0000 7fb01c690700 1 -- 172.21.15.195:0/2126079977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0140f5410 msgr2=0x7fb0140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:30.302 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.299+0000 7fb01c690700 1 --2- 172.21.15.195:0/2126079977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0140f5410 0x7fb0140f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fb004009fb0 tx=0x7fb004009b30).stop 2023-10-12T22:37:30.302 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.299+0000 7fb01c690700 1 -- 172.21.15.195:0/2126079977 shutdown_connections 2023-10-12T22:37:30.302 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.299+0000 7fb01c690700 1 --2- 172.21.15.195:0/2126079977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0140f5410 0x7fb0140f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:30.302 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.299+0000 7fb01c690700 1 -- 172.21.15.195:0/2126079977 >> 172.21.15.195:0/2126079977 conn(0x7fb0140f0b70 msgr2=0x7fb0140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:30.303 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.299+0000 7fb01c690700 1 -- 172.21.15.195:0/2126079977 shutdown_connections 2023-10-12T22:37:30.303 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.299+0000 7fb01c690700 1 -- 172.21.15.195:0/2126079977 wait complete. 2023-10-12T22:37:30.303 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.299+0000 7fb01c690700 1 Processor -- start 2023-10-12T22:37:30.303 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.299+0000 7fb01c690700 1 -- start start 2023-10-12T22:37:30.303 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.299+0000 7fb01c690700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0140f5410 0x7fb014106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:30.303 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.299+0000 7fb01c690700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb014106c20 con 0x7fb0140f5410 2023-10-12T22:37:30.304 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.300+0000 7fb01a42c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0140f5410 0x7fb014106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:30.304 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.300+0000 7fb01a42c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0140f5410 0x7fb014106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:49212/0 (socket says 172.21.15.195:49212) 2023-10-12T22:37:30.304 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.300+0000 7fb01a42c700 1 -- 172.21.15.195:0/1598278841 learned_addr learned my addr 172.21.15.195:0/1598278841 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:30.304 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.300+0000 7fb01a42c700 1 -- 172.21.15.195:0/1598278841 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb004005040 con 0x7fb0140f5410 2023-10-12T22:37:30.304 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.300+0000 7fb01a42c700 1 --2- 172.21.15.195:0/1598278841 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0140f5410 0x7fb014106730 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fb004013510 tx=0x7fb00400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:30.305 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.300+0000 7fb00affd700 1 -- 172.21.15.195:0/1598278841 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb004009320 con 0x7fb0140f5410 2023-10-12T22:37:30.305 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.300+0000 7fb01c690700 1 -- 172.21.15.195:0/1598278841 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb014106e20 con 0x7fb0140f5410 2023-10-12T22:37:30.305 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.300+0000 7fb00affd700 1 -- 172.21.15.195:0/1598278841 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb004013f00 con 0x7fb0140f5410 2023-10-12T22:37:30.305 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.300+0000 7fb00affd700 1 -- 172.21.15.195:0/1598278841 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb004014d90 con 0x7fb0140f5410 2023-10-12T22:37:30.305 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.300+0000 7fb01c690700 1 -- 172.21.15.195:0/1598278841 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0141072b0 con 0x7fb0140f5410 2023-10-12T22:37:30.306 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.301+0000 7fb00affd700 1 -- 172.21.15.195:0/1598278841 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0040095f0 con 0x7fb0140f5410 2023-10-12T22:37:30.307 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.301+0000 7fb01c690700 1 -- 172.21.15.195:0/1598278841 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb014045ad0 con 0x7fb0140f5410 2023-10-12T22:37:30.307 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.301+0000 7fb00affd700 1 --2- 172.21.15.195:0/1598278841 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb00002ff60 0x7fb000032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:30.307 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.301+0000 7fb00affd700 1 -- 172.21.15.195:0/1598278841 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb004041000 con 0x7fb0140f5410 2023-10-12T22:37:30.307 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.302+0000 7fb019c2b700 1 --2- 172.21.15.195:0/1598278841 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb00002ff60 0x7fb000032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:30.307 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.302+0000 7fb019c2b700 1 --2- 172.21.15.195:0/1598278841 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb00002ff60 0x7fb000032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fb0100096a0 tx=0x7fb010006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:30.308 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.306+0000 7fb00affd700 1 -- 172.21.15.195:0/1598278841 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb004014420 con 0x7fb0140f5410 2023-10-12T22:37:30.515 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.514+0000 7fb01c690700 1 -- 172.21.15.195:0/1598278841 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0140fae60 con 0x7fb0140f5410 2023-10-12T22:37:30.516 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.515+0000 7fb00affd700 1 -- 172.21.15.195:0/1598278841 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb00401d7f0 con 0x7fb0140f5410 2023-10-12T22:37:30.516 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:30.517 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:30.518 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.517+0000 7fb01c690700 1 -- 172.21.15.195:0/1598278841 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb00002ff60 msgr2=0x7fb000032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:30.518 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.517+0000 7fb01c690700 1 --2- 172.21.15.195:0/1598278841 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb00002ff60 0x7fb000032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fb0100096a0 tx=0x7fb010006b40).stop 2023-10-12T22:37:30.518 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.517+0000 7fb01c690700 1 -- 172.21.15.195:0/1598278841 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0140f5410 msgr2=0x7fb014106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:30.519 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.517+0000 7fb01c690700 1 --2- 172.21.15.195:0/1598278841 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0140f5410 0x7fb014106730 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fb004013510 tx=0x7fb00400b040).stop 2023-10-12T22:37:30.519 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.517+0000 7fb01c690700 1 -- 172.21.15.195:0/1598278841 shutdown_connections 2023-10-12T22:37:30.519 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.517+0000 7fb01c690700 1 --2- 172.21.15.195:0/1598278841 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb00002ff60 0x7fb000032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:30.519 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.517+0000 7fb01c690700 1 --2- 172.21.15.195:0/1598278841 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb0140f5410 0x7fb014106730 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:30.520 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.517+0000 7fb01c690700 1 -- 172.21.15.195:0/1598278841 >> 172.21.15.195:0/1598278841 conn(0x7fb0140f0b70 msgr2=0x7fb0140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:30.520 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.517+0000 7fb01c690700 1 -- 172.21.15.195:0/1598278841 shutdown_connections 2023-10-12T22:37:30.520 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:30.517+0000 7fb01c690700 1 -- 172.21.15.195:0/1598278841 wait complete. 2023-10-12T22:37:30.520 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:30.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:30 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:28.853583+0000 mgr.smithi131.efktma (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:30.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:30 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:30.516340+0000 mon.smithi131 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.195:0/1598278841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:31.939 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:31.939 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:32.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:30.853898+0000 mgr.smithi131.efktma (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:33.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.029+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/37656345 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d88060280 msgr2=0x7f9d880f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:33.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.029+0000 7f9d8dfdd700 1 --2- 172.21.15.195:0/37656345 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d88060280 0x7f9d880f90f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9d78009fb0 tx=0x7f9d78009b30).stop 2023-10-12T22:37:33.031 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.029+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/37656345 shutdown_connections 2023-10-12T22:37:33.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.029+0000 7f9d8dfdd700 1 --2- 172.21.15.195:0/37656345 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d88060280 0x7f9d880f90f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:33.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.029+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/37656345 >> 172.21.15.195:0/37656345 conn(0x7f9d880f0b70 msgr2=0x7f9d880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:33.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.029+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/37656345 shutdown_connections 2023-10-12T22:37:33.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.029+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/37656345 wait complete. 2023-10-12T22:37:33.032 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.030+0000 7f9d8dfdd700 1 Processor -- start 2023-10-12T22:37:33.033 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.030+0000 7f9d8dfdd700 1 -- start start 2023-10-12T22:37:33.033 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.030+0000 7f9d8dfdd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d88060280 0x7f9d881066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:33.033 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.030+0000 7f9d8dfdd700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88106bd0 con 0x7f9d88060280 2023-10-12T22:37:33.034 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.030+0000 7f9d877fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d88060280 0x7f9d881066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:33.035 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.030+0000 7f9d877fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d88060280 0x7f9d881066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:49238/0 (socket says 172.21.15.195:49238) 2023-10-12T22:37:33.035 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.030+0000 7f9d877fe700 1 -- 172.21.15.195:0/397717602 learned_addr learned my addr 172.21.15.195:0/397717602 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:33.035 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.030+0000 7f9d877fe700 1 -- 172.21.15.195:0/397717602 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d78005040 con 0x7f9d88060280 2023-10-12T22:37:33.035 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.030+0000 7f9d877fe700 1 --2- 172.21.15.195:0/397717602 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d88060280 0x7f9d881066e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f9d78013510 tx=0x7f9d7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:33.036 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.031+0000 7f9d7ffff700 1 -- 172.21.15.195:0/397717602 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d78009320 con 0x7f9d88060280 2023-10-12T22:37:33.036 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.031+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/397717602 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d88106dd0 con 0x7f9d88060280 2023-10-12T22:37:33.036 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.031+0000 7f9d7ffff700 1 -- 172.21.15.195:0/397717602 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d78013f00 con 0x7f9d88060280 2023-10-12T22:37:33.036 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.031+0000 7f9d7ffff700 1 -- 172.21.15.195:0/397717602 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d78014dc0 con 0x7f9d88060280 2023-10-12T22:37:33.036 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.031+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/397717602 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d88107260 con 0x7f9d88060280 2023-10-12T22:37:33.037 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.031+0000 7f9d7ffff700 1 -- 172.21.15.195:0/397717602 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d780095f0 con 0x7f9d88060280 2023-10-12T22:37:33.037 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.032+0000 7f9d7ffff700 1 --2- 172.21.15.195:0/397717602 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9d7402ff60 0x7f9d74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:33.037 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.032+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/397717602 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d88045ad0 con 0x7f9d88060280 2023-10-12T22:37:33.037 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.032+0000 7f9d7ffff700 1 -- 172.21.15.195:0/397717602 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d78041100 con 0x7f9d88060280 2023-10-12T22:37:33.037 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.032+0000 7f9d86ffd700 1 --2- 172.21.15.195:0/397717602 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9d7402ff60 0x7f9d74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:33.037 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.032+0000 7f9d86ffd700 1 --2- 172.21.15.195:0/397717602 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9d7402ff60 0x7f9d74032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9d700096a0 tx=0x7f9d70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:33.038 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.036+0000 7f9d7ffff700 1 -- 172.21.15.195:0/397717602 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d78014420 con 0x7f9d88060280 2023-10-12T22:37:33.246 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.242+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/397717602 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d88108110 con 0x7f9d88060280 2023-10-12T22:37:33.247 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.243+0000 7f9d7ffff700 1 -- 172.21.15.195:0/397717602 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d78010230 con 0x7f9d88060280 2023-10-12T22:37:33.247 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:33.248 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:33.248 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.244+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/397717602 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9d7402ff60 msgr2=0x7f9d74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:33.249 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.244+0000 7f9d8dfdd700 1 --2- 172.21.15.195:0/397717602 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9d7402ff60 0x7f9d74032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9d700096a0 tx=0x7f9d70006b40).stop 2023-10-12T22:37:33.249 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.244+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/397717602 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d88060280 msgr2=0x7f9d881066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:33.249 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.244+0000 7f9d8dfdd700 1 --2- 172.21.15.195:0/397717602 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d88060280 0x7f9d881066e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f9d78013510 tx=0x7f9d7800b040).stop 2023-10-12T22:37:33.249 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.244+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/397717602 shutdown_connections 2023-10-12T22:37:33.249 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.244+0000 7f9d8dfdd700 1 --2- 172.21.15.195:0/397717602 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9d7402ff60 0x7f9d74032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:33.250 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.244+0000 7f9d8dfdd700 1 --2- 172.21.15.195:0/397717602 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9d88060280 0x7f9d881066e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:33.250 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.244+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/397717602 >> 172.21.15.195:0/397717602 conn(0x7f9d880f0b70 msgr2=0x7f9d880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:33.250 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.244+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/397717602 shutdown_connections 2023-10-12T22:37:33.250 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:33.244+0000 7f9d8dfdd700 1 -- 172.21.15.195:0/397717602 wait complete. 2023-10-12T22:37:33.250 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:33.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:33 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:33.244077+0000 mon.smithi131 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.195:0/397717602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:34.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:34.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:34.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:32.854255+0000 mgr.smithi131.efktma (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:35.805 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.803+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2742700880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a780f6a80 msgr2=0x7f3a780f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:35.805 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.803+0000 7f3a7ee62700 1 --2- 172.21.15.195:0/2742700880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a780f6a80 0x7f3a780f6ea0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f3a74009fb0 tx=0x7f3a74009b30).stop 2023-10-12T22:37:35.805 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.803+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2742700880 shutdown_connections 2023-10-12T22:37:35.805 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.803+0000 7f3a7ee62700 1 --2- 172.21.15.195:0/2742700880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a780f6a80 0x7f3a780f6ea0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.803+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2742700880 >> 172.21.15.195:0/2742700880 conn(0x7f3a780f02e0 msgr2=0x7f3a780f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.803+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2742700880 shutdown_connections 2023-10-12T22:37:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.803+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2742700880 wait complete. 2023-10-12T22:37:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.804+0000 7f3a7ee62700 1 Processor -- start 2023-10-12T22:37:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.804+0000 7f3a7ee62700 1 -- start start 2023-10-12T22:37:35.807 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.804+0000 7f3a7ee62700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a780f6a80 0x7f3a78109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:35.807 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.804+0000 7f3a7ee62700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a780f5330 con 0x7f3a780f6a80 2023-10-12T22:37:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.804+0000 7f3a7cbfe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a780f6a80 0x7f3a78109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.804+0000 7f3a7cbfe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a780f6a80 0x7f3a78109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:49254/0 (socket says 172.21.15.195:49254) 2023-10-12T22:37:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.804+0000 7f3a7cbfe700 1 -- 172.21.15.195:0/2350592396 learned_addr learned my addr 172.21.15.195:0/2350592396 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.805+0000 7f3a7cbfe700 1 -- 172.21.15.195:0/2350592396 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a74005040 con 0x7f3a780f6a80 2023-10-12T22:37:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.805+0000 7f3a7cbfe700 1 --2- 172.21.15.195:0/2350592396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a780f6a80 0x7f3a78109410 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f3a74013510 tx=0x7f3a7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.805+0000 7f3a6d7fa700 1 -- 172.21.15.195:0/2350592396 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a74009320 con 0x7f3a780f6a80 2023-10-12T22:37:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.805+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2350592396 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a780f5530 con 0x7f3a780f6a80 2023-10-12T22:37:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.805+0000 7f3a6d7fa700 1 -- 172.21.15.195:0/2350592396 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a74013f00 con 0x7f3a780f6a80 2023-10-12T22:37:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.805+0000 7f3a6d7fa700 1 -- 172.21.15.195:0/2350592396 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a74014dc0 con 0x7f3a780f6a80 2023-10-12T22:37:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.805+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2350592396 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a78109ac0 con 0x7f3a780f6a80 2023-10-12T22:37:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.806+0000 7f3a6d7fa700 1 -- 172.21.15.195:0/2350592396 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a74014420 con 0x7f3a780f6a80 2023-10-12T22:37:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.806+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2350592396 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a78045ad0 con 0x7f3a780f6a80 2023-10-12T22:37:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.806+0000 7f3a6d7fa700 1 --2- 172.21.15.195:0/2350592396 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3a6002ff60 0x7f3a60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.806+0000 7f3a6d7fa700 1 -- 172.21.15.195:0/2350592396 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a74041030 con 0x7f3a780f6a80 2023-10-12T22:37:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.806+0000 7f3a6ffff700 1 --2- 172.21.15.195:0/2350592396 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3a6002ff60 0x7f3a60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.806+0000 7f3a6ffff700 1 --2- 172.21.15.195:0/2350592396 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3a6002ff60 0x7f3a60032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f3a680096a0 tx=0x7f3a68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:35.810+0000 7f3a6d7fa700 1 -- 172.21.15.195:0/2350592396 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a740095f0 con 0x7f3a780f6a80 2023-10-12T22:37:36.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:36.014+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2350592396 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a7810a2e0 con 0x7f3a780f6a80 2023-10-12T22:37:36.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:36.015+0000 7f3a6d7fa700 1 -- 172.21.15.195:0/2350592396 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3a7401cb10 con 0x7f3a780f6a80 2023-10-12T22:37:36.017 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:36.017 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:36.018 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:36.016+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2350592396 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3a6002ff60 msgr2=0x7f3a60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:36.018 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:36.016+0000 7f3a7ee62700 1 --2- 172.21.15.195:0/2350592396 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3a6002ff60 0x7f3a60032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f3a680096a0 tx=0x7f3a68006b40).stop 2023-10-12T22:37:36.018 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:36.016+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2350592396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a780f6a80 msgr2=0x7f3a78109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:36.018 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:36.016+0000 7f3a7ee62700 1 --2- 172.21.15.195:0/2350592396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a780f6a80 0x7f3a78109410 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f3a74013510 tx=0x7f3a7400b040).stop 2023-10-12T22:37:36.018 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:36.017+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2350592396 shutdown_connections 2023-10-12T22:37:36.019 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:36.017+0000 7f3a7ee62700 1 --2- 172.21.15.195:0/2350592396 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3a6002ff60 0x7f3a60032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:36.019 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:36.017+0000 7f3a7ee62700 1 --2- 172.21.15.195:0/2350592396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a780f6a80 0x7f3a78109410 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:36.019 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:36.017+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2350592396 >> 172.21.15.195:0/2350592396 conn(0x7f3a780f02e0 msgr2=0x7f3a780f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:36.019 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:36.017+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2350592396 shutdown_connections 2023-10-12T22:37:36.019 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:36.017+0000 7f3a7ee62700 1 -- 172.21.15.195:0/2350592396 wait complete. 2023-10-12T22:37:36.020 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:36.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:36 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:34.854572+0000 mgr.smithi131.efktma (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:36.990 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:36 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:36.016460+0000 mon.smithi131 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.195:0/2350592396' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:37.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:37.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:38.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.574+0000 7f626b0c6700 1 -- 172.21.15.195:0/3372539546 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62640f5410 msgr2=0x7f62640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:38.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.574+0000 7f626b0c6700 1 --2- 172.21.15.195:0/3372539546 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62640f5410 0x7f62640f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f6258009fb0 tx=0x7f6258009b30).stop 2023-10-12T22:37:38.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.574+0000 7f626b0c6700 1 -- 172.21.15.195:0/3372539546 shutdown_connections 2023-10-12T22:37:38.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.574+0000 7f626b0c6700 1 --2- 172.21.15.195:0/3372539546 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62640f5410 0x7f62640f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:38.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.574+0000 7f626b0c6700 1 -- 172.21.15.195:0/3372539546 >> 172.21.15.195:0/3372539546 conn(0x7f62640f0b70 msgr2=0x7f62640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:38.578 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.574+0000 7f626b0c6700 1 -- 172.21.15.195:0/3372539546 shutdown_connections 2023-10-12T22:37:38.578 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.574+0000 7f626b0c6700 1 -- 172.21.15.195:0/3372539546 wait complete. 2023-10-12T22:37:38.578 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.575+0000 7f626b0c6700 1 Processor -- start 2023-10-12T22:37:38.578 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.575+0000 7f626b0c6700 1 -- start start 2023-10-12T22:37:38.578 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.575+0000 7f626b0c6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62640f5410 0x7f6264109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:38.579 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.575+0000 7f626b0c6700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6264060240 con 0x7f62640f5410 2023-10-12T22:37:38.579 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.575+0000 7f6268e62700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62640f5410 0x7f6264109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:38.579 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.575+0000 7f6268e62700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62640f5410 0x7f6264109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:57806/0 (socket says 172.21.15.195:57806) 2023-10-12T22:37:38.579 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.575+0000 7f6268e62700 1 -- 172.21.15.195:0/4159120180 learned_addr learned my addr 172.21.15.195:0/4159120180 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:38.579 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.576+0000 7f6268e62700 1 -- 172.21.15.195:0/4159120180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6258005040 con 0x7f62640f5410 2023-10-12T22:37:38.581 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.576+0000 7f6268e62700 1 --2- 172.21.15.195:0/4159120180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62640f5410 0x7f6264109430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f6258013510 tx=0x7f625800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:38.581 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.576+0000 7f62617fa700 1 -- 172.21.15.195:0/4159120180 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6258009320 con 0x7f62640f5410 2023-10-12T22:37:38.581 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.576+0000 7f626b0c6700 1 -- 172.21.15.195:0/4159120180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6264060440 con 0x7f62640f5410 2023-10-12T22:37:38.581 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.576+0000 7f62617fa700 1 -- 172.21.15.195:0/4159120180 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6258013f00 con 0x7f62640f5410 2023-10-12T22:37:38.582 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.576+0000 7f62617fa700 1 -- 172.21.15.195:0/4159120180 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6258014d90 con 0x7f62640f5410 2023-10-12T22:37:38.582 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.576+0000 7f626b0c6700 1 -- 172.21.15.195:0/4159120180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6264109ae0 con 0x7f62640f5410 2023-10-12T22:37:38.582 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.577+0000 7f62617fa700 1 -- 172.21.15.195:0/4159120180 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6258014420 con 0x7f62640f5410 2023-10-12T22:37:38.582 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.577+0000 7f626b0c6700 1 -- 172.21.15.195:0/4159120180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6264045ad0 con 0x7f62640f5410 2023-10-12T22:37:38.582 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.577+0000 7f62617fa700 1 --2- 172.21.15.195:0/4159120180 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f625402ff60 0x7f6254032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:38.583 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.577+0000 7f62617fa700 1 -- 172.21.15.195:0/4159120180 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62580410c0 con 0x7f62640f5410 2023-10-12T22:37:38.583 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.577+0000 7f6263fff700 1 --2- 172.21.15.195:0/4159120180 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f625402ff60 0x7f6254032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:38.583 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.578+0000 7f6263fff700 1 --2- 172.21.15.195:0/4159120180 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f625402ff60 0x7f6254032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f62500096a0 tx=0x7f6250006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:38.583 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.582+0000 7f62617fa700 1 -- 172.21.15.195:0/4159120180 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62580095f0 con 0x7f62640f5410 2023-10-12T22:37:38.788 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.787+0000 7f626b0c6700 1 -- 172.21.15.195:0/4159120180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62640fae60 con 0x7f62640f5410 2023-10-12T22:37:38.789 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.788+0000 7f62617fa700 1 -- 172.21.15.195:0/4159120180 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f625801db70 con 0x7f62640f5410 2023-10-12T22:37:38.789 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:38.789 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:38.790 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.789+0000 7f626b0c6700 1 -- 172.21.15.195:0/4159120180 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f625402ff60 msgr2=0x7f6254032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:38.790 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.789+0000 7f626b0c6700 1 --2- 172.21.15.195:0/4159120180 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f625402ff60 0x7f6254032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f62500096a0 tx=0x7f6250006b40).stop 2023-10-12T22:37:38.791 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.789+0000 7f626b0c6700 1 -- 172.21.15.195:0/4159120180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62640f5410 msgr2=0x7f6264109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:38.791 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.789+0000 7f626b0c6700 1 --2- 172.21.15.195:0/4159120180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62640f5410 0x7f6264109430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f6258013510 tx=0x7f625800b040).stop 2023-10-12T22:37:38.791 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.789+0000 7f626b0c6700 1 -- 172.21.15.195:0/4159120180 shutdown_connections 2023-10-12T22:37:38.791 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.789+0000 7f626b0c6700 1 --2- 172.21.15.195:0/4159120180 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f625402ff60 0x7f6254032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:38.792 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.789+0000 7f626b0c6700 1 --2- 172.21.15.195:0/4159120180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62640f5410 0x7f6264109430 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:38.792 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.789+0000 7f626b0c6700 1 -- 172.21.15.195:0/4159120180 >> 172.21.15.195:0/4159120180 conn(0x7f62640f0b70 msgr2=0x7f62640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:38.792 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.789+0000 7f626b0c6700 1 -- 172.21.15.195:0/4159120180 shutdown_connections 2023-10-12T22:37:38.792 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:38.789+0000 7f626b0c6700 1 -- 172.21.15.195:0/4159120180 wait complete. 2023-10-12T22:37:38.793 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:38.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:38 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:36.854970+0000 mgr.smithi131.efktma (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:39.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:39 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:38.788997+0000 mon.smithi131 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.195:0/4159120180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:40.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:40.214 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:40.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:40 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:38.855334+0000 mgr.smithi131.efktma (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:41.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.405+0000 7fc12fe98700 1 -- 172.21.15.195:0/3518166356 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1280f5410 msgr2=0x7fc1280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:41.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.405+0000 7fc12fe98700 1 --2- 172.21.15.195:0/3518166356 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1280f5410 0x7fc1280f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fc118009fb0 tx=0x7fc118009b30).stop 2023-10-12T22:37:41.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.405+0000 7fc12fe98700 1 -- 172.21.15.195:0/3518166356 shutdown_connections 2023-10-12T22:37:41.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.405+0000 7fc12fe98700 1 --2- 172.21.15.195:0/3518166356 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1280f5410 0x7fc1280f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:41.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.405+0000 7fc12fe98700 1 -- 172.21.15.195:0/3518166356 >> 172.21.15.195:0/3518166356 conn(0x7fc1280f0b70 msgr2=0x7fc1280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:41.408 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.405+0000 7fc12fe98700 1 -- 172.21.15.195:0/3518166356 shutdown_connections 2023-10-12T22:37:41.408 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.405+0000 7fc12fe98700 1 -- 172.21.15.195:0/3518166356 wait complete. 2023-10-12T22:37:41.410 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.406+0000 7fc12fe98700 1 Processor -- start 2023-10-12T22:37:41.410 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.406+0000 7fc12fe98700 1 -- start start 2023-10-12T22:37:41.410 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.406+0000 7fc12fe98700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1280f5410 0x7fc128105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:41.411 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.406+0000 7fc12fe98700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc128105af0 con 0x7fc1280f5410 2023-10-12T22:37:41.411 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.406+0000 7fc12dc34700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1280f5410 0x7fc128105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:41.411 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.406+0000 7fc12dc34700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1280f5410 0x7fc128105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:57820/0 (socket says 172.21.15.195:57820) 2023-10-12T22:37:41.411 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.406+0000 7fc12dc34700 1 -- 172.21.15.195:0/3542161382 learned_addr learned my addr 172.21.15.195:0/3542161382 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:41.411 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.407+0000 7fc12dc34700 1 -- 172.21.15.195:0/3542161382 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc118005040 con 0x7fc1280f5410 2023-10-12T22:37:41.412 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.407+0000 7fc12dc34700 1 --2- 172.21.15.195:0/3542161382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1280f5410 0x7fc128105600 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fc118013510 tx=0x7fc11800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:41.412 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.407+0000 7fc11e7fc700 1 -- 172.21.15.195:0/3542161382 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc118009320 con 0x7fc1280f5410 2023-10-12T22:37:41.412 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.407+0000 7fc12fe98700 1 -- 172.21.15.195:0/3542161382 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc128105cf0 con 0x7fc1280f5410 2023-10-12T22:37:41.412 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.407+0000 7fc11e7fc700 1 -- 172.21.15.195:0/3542161382 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc118013f00 con 0x7fc1280f5410 2023-10-12T22:37:41.412 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.407+0000 7fc11e7fc700 1 -- 172.21.15.195:0/3542161382 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc118014d90 con 0x7fc1280f5410 2023-10-12T22:37:41.413 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.407+0000 7fc12fe98700 1 -- 172.21.15.195:0/3542161382 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc128106180 con 0x7fc1280f5410 2023-10-12T22:37:41.413 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.408+0000 7fc11e7fc700 1 -- 172.21.15.195:0/3542161382 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1180095f0 con 0x7fc1280f5410 2023-10-12T22:37:41.413 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.408+0000 7fc12fe98700 1 -- 172.21.15.195:0/3542161382 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc128045ad0 con 0x7fc1280f5410 2023-10-12T22:37:41.413 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.408+0000 7fc11e7fc700 1 --2- 172.21.15.195:0/3542161382 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc11402ff60 0x7fc114032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:41.413 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.408+0000 7fc11e7fc700 1 -- 172.21.15.195:0/3542161382 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc118040fe0 con 0x7fc1280f5410 2023-10-12T22:37:41.413 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.408+0000 7fc12d433700 1 --2- 172.21.15.195:0/3542161382 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc11402ff60 0x7fc114032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:41.414 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.409+0000 7fc12d433700 1 --2- 172.21.15.195:0/3542161382 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc11402ff60 0x7fc114032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fc1240096a0 tx=0x7fc124006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:41.414 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.412+0000 7fc11e7fc700 1 -- 172.21.15.195:0/3542161382 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc11801d7f0 con 0x7fc1280f5410 2023-10-12T22:37:41.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.617+0000 7fc12fe98700 1 -- 172.21.15.195:0/3542161382 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1280fae60 con 0x7fc1280f5410 2023-10-12T22:37:41.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.617+0000 7fc11e7fc700 1 -- 172.21.15.195:0/3542161382 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc118010230 con 0x7fc1280f5410 2023-10-12T22:37:41.619 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:41.619 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:41.620 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.619+0000 7fc12fe98700 1 -- 172.21.15.195:0/3542161382 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc11402ff60 msgr2=0x7fc114032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:41.621 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.619+0000 7fc12fe98700 1 --2- 172.21.15.195:0/3542161382 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc11402ff60 0x7fc114032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fc1240096a0 tx=0x7fc124006b40).stop 2023-10-12T22:37:41.621 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.619+0000 7fc12fe98700 1 -- 172.21.15.195:0/3542161382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1280f5410 msgr2=0x7fc128105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:41.621 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.619+0000 7fc12fe98700 1 --2- 172.21.15.195:0/3542161382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1280f5410 0x7fc128105600 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fc118013510 tx=0x7fc11800b040).stop 2023-10-12T22:37:41.621 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.619+0000 7fc12fe98700 1 -- 172.21.15.195:0/3542161382 shutdown_connections 2023-10-12T22:37:41.621 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.619+0000 7fc12fe98700 1 --2- 172.21.15.195:0/3542161382 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc11402ff60 0x7fc114032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:41.622 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.619+0000 7fc12fe98700 1 --2- 172.21.15.195:0/3542161382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1280f5410 0x7fc128105600 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:41.622 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.619+0000 7fc12fe98700 1 -- 172.21.15.195:0/3542161382 >> 172.21.15.195:0/3542161382 conn(0x7fc1280f0b70 msgr2=0x7fc1280f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:41.622 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.619+0000 7fc12fe98700 1 -- 172.21.15.195:0/3542161382 shutdown_connections 2023-10-12T22:37:41.622 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:41.619+0000 7fc12fe98700 1 -- 172.21.15.195:0/3542161382 wait complete. 2023-10-12T22:37:41.623 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:42.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:42 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:40.855604+0000 mgr.smithi131.efktma (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:42.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:42 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:41.618684+0000 mon.smithi131 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.195:0/3542161382' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:43.005 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:43.005 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:44.189 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.188+0000 7f92ba421700 1 -- 172.21.15.195:0/2856591395 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b40f6510 msgr2=0x7f92b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:44.190 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.188+0000 7f92ba421700 1 --2- 172.21.15.195:0/2856591395 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b40f6510 0x7f92b40f6930 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f929c009fb0 tx=0x7f929c009b30).stop 2023-10-12T22:37:44.190 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.188+0000 7f92ba421700 1 -- 172.21.15.195:0/2856591395 shutdown_connections 2023-10-12T22:37:44.190 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.188+0000 7f92ba421700 1 --2- 172.21.15.195:0/2856591395 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b40f6510 0x7f92b40f6930 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:44.191 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.188+0000 7f92ba421700 1 -- 172.21.15.195:0/2856591395 >> 172.21.15.195:0/2856591395 conn(0x7f92b40f1c70 msgr2=0x7f92b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:44.191 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.188+0000 7f92ba421700 1 -- 172.21.15.195:0/2856591395 shutdown_connections 2023-10-12T22:37:44.191 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.188+0000 7f92ba421700 1 -- 172.21.15.195:0/2856591395 wait complete. 2023-10-12T22:37:44.191 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.189+0000 7f92ba421700 1 Processor -- start 2023-10-12T22:37:44.193 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.189+0000 7f92ba421700 1 -- start start 2023-10-12T22:37:44.193 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.189+0000 7f92ba421700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b40f6510 0x7f92b4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:44.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.189+0000 7f92ba421700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b4106c20 con 0x7f92b40f6510 2023-10-12T22:37:44.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.189+0000 7f92b3fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b40f6510 0x7f92b4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:44.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.189+0000 7f92b3fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b40f6510 0x7f92b4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:57834/0 (socket says 172.21.15.195:57834) 2023-10-12T22:37:44.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.189+0000 7f92b3fff700 1 -- 172.21.15.195:0/2810732415 learned_addr learned my addr 172.21.15.195:0/2810732415 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:44.194 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.190+0000 7f92b3fff700 1 -- 172.21.15.195:0/2810732415 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c005040 con 0x7f92b40f6510 2023-10-12T22:37:44.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.190+0000 7f92b3fff700 1 --2- 172.21.15.195:0/2810732415 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b40f6510 0x7f92b4106730 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f929c013510 tx=0x7f929c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:44.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.190+0000 7f92b0ff9700 1 -- 172.21.15.195:0/2810732415 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f929c009320 con 0x7f92b40f6510 2023-10-12T22:37:44.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.190+0000 7f92ba421700 1 -- 172.21.15.195:0/2810732415 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b4106e20 con 0x7f92b40f6510 2023-10-12T22:37:44.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.190+0000 7f92b0ff9700 1 -- 172.21.15.195:0/2810732415 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f929c013f00 con 0x7f92b40f6510 2023-10-12T22:37:44.195 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.190+0000 7f92b0ff9700 1 -- 172.21.15.195:0/2810732415 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f929c014d90 con 0x7f92b40f6510 2023-10-12T22:37:44.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.190+0000 7f92ba421700 1 -- 172.21.15.195:0/2810732415 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b41072b0 con 0x7f92b40f6510 2023-10-12T22:37:44.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.191+0000 7f92b0ff9700 1 -- 172.21.15.195:0/2810732415 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f929c0263a0 con 0x7f92b40f6510 2023-10-12T22:37:44.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.191+0000 7f92b0ff9700 1 --2- 172.21.15.195:0/2810732415 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f92a002ffb0 0x7f92a0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:44.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.191+0000 7f92b0ff9700 1 -- 172.21.15.195:0/2810732415 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f929c0410a0 con 0x7f92b40f6510 2023-10-12T22:37:44.196 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.191+0000 7f92ba421700 1 -- 172.21.15.195:0/2810732415 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b4045ad0 con 0x7f92b40f6510 2023-10-12T22:37:44.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.191+0000 7f92b37fe700 1 --2- 172.21.15.195:0/2810732415 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f92a002ffb0 0x7f92a0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:44.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.192+0000 7f92b37fe700 1 --2- 172.21.15.195:0/2810732415 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f92a002ffb0 0x7f92a0032470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f92a40096a0 tx=0x7f92a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:44.197 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.196+0000 7f92b0ff9700 1 -- 172.21.15.195:0/2810732415 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f929c012920 con 0x7f92b40f6510 2023-10-12T22:37:44.400 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.399+0000 7f92ba421700 1 -- 172.21.15.195:0/2810732415 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92b4107fd0 con 0x7f92b40f6510 2023-10-12T22:37:44.401 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.399+0000 7f92b0ff9700 1 -- 172.21.15.195:0/2810732415 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f929c02b020 con 0x7f92b40f6510 2023-10-12T22:37:44.401 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:44.401 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:44.402 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.401+0000 7f92ba421700 1 -- 172.21.15.195:0/2810732415 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f92a002ffb0 msgr2=0x7f92a0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:44.402 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.401+0000 7f92ba421700 1 --2- 172.21.15.195:0/2810732415 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f92a002ffb0 0x7f92a0032470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f92a40096a0 tx=0x7f92a4006b40).stop 2023-10-12T22:37:44.403 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.401+0000 7f92ba421700 1 -- 172.21.15.195:0/2810732415 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b40f6510 msgr2=0x7f92b4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:44.403 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.401+0000 7f92ba421700 1 --2- 172.21.15.195:0/2810732415 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b40f6510 0x7f92b4106730 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f929c013510 tx=0x7f929c00b040).stop 2023-10-12T22:37:44.403 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.401+0000 7f92ba421700 1 -- 172.21.15.195:0/2810732415 shutdown_connections 2023-10-12T22:37:44.403 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.401+0000 7f92ba421700 1 --2- 172.21.15.195:0/2810732415 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f92a002ffb0 0x7f92a0032470 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:44.403 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.401+0000 7f92ba421700 1 --2- 172.21.15.195:0/2810732415 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b40f6510 0x7f92b4106730 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:44.403 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.401+0000 7f92ba421700 1 -- 172.21.15.195:0/2810732415 >> 172.21.15.195:0/2810732415 conn(0x7f92b40f1c70 msgr2=0x7f92b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:44.404 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.401+0000 7f92ba421700 1 -- 172.21.15.195:0/2810732415 shutdown_connections 2023-10-12T22:37:44.404 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:44.401+0000 7f92ba421700 1 -- 172.21.15.195:0/2810732415 wait complete. 2023-10-12T22:37:44.404 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:44.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:44 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:42.855938+0000 mgr.smithi131.efktma (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:44.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:44 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:44.400785+0000 mon.smithi131 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.195:0/2810732415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:45.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:45.763 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:46.950 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.948+0000 7f5799d9b700 1 -- 172.21.15.195:0/3439540415 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57900138a0 con 0x7f579c0f5410 2023-10-12T22:37:46.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.949+0000 7f57a11c7700 1 -- 172.21.15.195:0/3439540415 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f579c0f5410 msgr2=0x7f579c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:46.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.949+0000 7f57a11c7700 1 --2- 172.21.15.195:0/3439540415 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f579c0f5410 0x7f579c0f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f5790009fb0 tx=0x7f5790009b30).stop 2023-10-12T22:37:46.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.949+0000 7f57a11c7700 1 -- 172.21.15.195:0/3439540415 shutdown_connections 2023-10-12T22:37:46.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.949+0000 7f57a11c7700 1 --2- 172.21.15.195:0/3439540415 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f579c0f5410 0x7f579c0f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:46.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.949+0000 7f57a11c7700 1 -- 172.21.15.195:0/3439540415 >> 172.21.15.195:0/3439540415 conn(0x7f579c0f0b70 msgr2=0x7f579c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:46.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.949+0000 7f57a11c7700 1 -- 172.21.15.195:0/3439540415 shutdown_connections 2023-10-12T22:37:46.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.949+0000 7f57a11c7700 1 -- 172.21.15.195:0/3439540415 wait complete. 2023-10-12T22:37:46.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.950+0000 7f57a11c7700 1 Processor -- start 2023-10-12T22:37:46.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.950+0000 7f57a11c7700 1 -- start start 2023-10-12T22:37:46.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.950+0000 7f57a11c7700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f579c0f5410 0x7f579c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:46.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.950+0000 7f57a11c7700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579c106bd0 con 0x7f579c0f5410 2023-10-12T22:37:46.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.950+0000 7f579ad9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f579c0f5410 0x7f579c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:46.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.950+0000 7f579ad9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f579c0f5410 0x7f579c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:57856/0 (socket says 172.21.15.195:57856) 2023-10-12T22:37:46.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.950+0000 7f579ad9d700 1 -- 172.21.15.195:0/455475562 learned_addr learned my addr 172.21.15.195:0/455475562 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:46.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.950+0000 7f579ad9d700 1 -- 172.21.15.195:0/455475562 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5790005040 con 0x7f579c0f5410 2023-10-12T22:37:46.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.950+0000 7f579ad9d700 1 --2- 172.21.15.195:0/455475562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f579c0f5410 0x7f579c1066e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f5790013510 tx=0x7f579000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:46.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.951+0000 7f57837fe700 1 -- 172.21.15.195:0/455475562 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5790009320 con 0x7f579c0f5410 2023-10-12T22:37:46.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.951+0000 7f57a11c7700 1 -- 172.21.15.195:0/455475562 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f579c106dd0 con 0x7f579c0f5410 2023-10-12T22:37:46.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.951+0000 7f57837fe700 1 -- 172.21.15.195:0/455475562 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5790013f00 con 0x7f579c0f5410 2023-10-12T22:37:46.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.951+0000 7f57837fe700 1 -- 172.21.15.195:0/455475562 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5790014d90 con 0x7f579c0f5410 2023-10-12T22:37:46.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.951+0000 7f57a11c7700 1 -- 172.21.15.195:0/455475562 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f579c107260 con 0x7f579c0f5410 2023-10-12T22:37:46.956 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.952+0000 7f57837fe700 1 -- 172.21.15.195:0/455475562 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57900095f0 con 0x7f579c0f5410 2023-10-12T22:37:46.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.952+0000 7f57a11c7700 1 -- 172.21.15.195:0/455475562 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f579c045ad0 con 0x7f579c0f5410 2023-10-12T22:37:46.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.952+0000 7f57837fe700 1 --2- 172.21.15.195:0/455475562 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f578402ff60 0x7f5784032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:46.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.952+0000 7f57837fe700 1 -- 172.21.15.195:0/455475562 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5790041080 con 0x7f579c0f5410 2023-10-12T22:37:46.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.952+0000 7f579a59c700 1 --2- 172.21.15.195:0/455475562 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f578402ff60 0x7f5784032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:46.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.953+0000 7f579a59c700 1 --2- 172.21.15.195:0/455475562 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f578402ff60 0x7f5784032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f578c0096a0 tx=0x7f578c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:46.958 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:46.956+0000 7f57837fe700 1 -- 172.21.15.195:0/455475562 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5790014420 con 0x7f579c0f5410 2023-10-12T22:37:46.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:46 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:44.856304+0000 mgr.smithi131.efktma (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:47.162 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:47.161+0000 7f57a11c7700 1 -- 172.21.15.195:0/455475562 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f579c0fae60 con 0x7f579c0f5410 2023-10-12T22:37:47.163 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:47.162+0000 7f57837fe700 1 -- 172.21.15.195:0/455475562 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5790010230 con 0x7f579c0f5410 2023-10-12T22:37:47.164 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:47.164 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:47.165 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:47.163+0000 7f57a11c7700 1 -- 172.21.15.195:0/455475562 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f578402ff60 msgr2=0x7f5784032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:47.165 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:47.163+0000 7f57a11c7700 1 --2- 172.21.15.195:0/455475562 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f578402ff60 0x7f5784032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f578c0096a0 tx=0x7f578c006b40).stop 2023-10-12T22:37:47.165 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:47.163+0000 7f57a11c7700 1 -- 172.21.15.195:0/455475562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f579c0f5410 msgr2=0x7f579c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:47.166 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:47.163+0000 7f57a11c7700 1 --2- 172.21.15.195:0/455475562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f579c0f5410 0x7f579c1066e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f5790013510 tx=0x7f579000b040).stop 2023-10-12T22:37:47.166 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:47.163+0000 7f57a11c7700 1 -- 172.21.15.195:0/455475562 shutdown_connections 2023-10-12T22:37:47.166 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:47.163+0000 7f57a11c7700 1 --2- 172.21.15.195:0/455475562 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f578402ff60 0x7f5784032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:47.166 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:47.163+0000 7f57a11c7700 1 --2- 172.21.15.195:0/455475562 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f579c0f5410 0x7f579c1066e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:47.166 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:47.163+0000 7f57a11c7700 1 -- 172.21.15.195:0/455475562 >> 172.21.15.195:0/455475562 conn(0x7f579c0f0b70 msgr2=0x7f579c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:47.167 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:47.164+0000 7f57a11c7700 1 -- 172.21.15.195:0/455475562 shutdown_connections 2023-10-12T22:37:47.167 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:47.164+0000 7f57a11c7700 1 -- 172.21.15.195:0/455475562 wait complete. 2023-10-12T22:37:47.167 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:47.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:47 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:47.163419+0000 mon.smithi131 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.195:0/455475562' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:48.583 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:48.584 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:48.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:48 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:46.856622+0000 mgr.smithi131.efktma (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:49.713 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.711+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/3790146868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2d80f6a10 msgr2=0x7ff2d80f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:49.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.711+0000 7ff2dd7c7700 1 --2- 172.21.15.195:0/3790146868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2d80f6a10 0x7ff2d80f6e30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7ff2cc009fb0 tx=0x7ff2cc009b30).stop 2023-10-12T22:37:49.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.711+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/3790146868 shutdown_connections 2023-10-12T22:37:49.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.711+0000 7ff2dd7c7700 1 --2- 172.21.15.195:0/3790146868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2d80f6a10 0x7ff2d80f6e30 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:49.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.711+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/3790146868 >> 172.21.15.195:0/3790146868 conn(0x7ff2d80f02e0 msgr2=0x7ff2d80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:49.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.711+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/3790146868 shutdown_connections 2023-10-12T22:37:49.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.711+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/3790146868 wait complete. 2023-10-12T22:37:49.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.712+0000 7ff2dd7c7700 1 Processor -- start 2023-10-12T22:37:49.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.712+0000 7ff2dd7c7700 1 -- start start 2023-10-12T22:37:49.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.712+0000 7ff2dd7c7700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2d80f6a10 0x7ff2d8106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:49.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.712+0000 7ff2dd7c7700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d8106bf0 con 0x7ff2d80f6a10 2023-10-12T22:37:49.717 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.713+0000 7ff2d6ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2d80f6a10 0x7ff2d8106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:49.717 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.713+0000 7ff2d6ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2d80f6a10 0x7ff2d8106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:37780/0 (socket says 172.21.15.195:37780) 2023-10-12T22:37:49.717 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.713+0000 7ff2d6ffd700 1 -- 172.21.15.195:0/203624633 learned_addr learned my addr 172.21.15.195:0/203624633 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:49.717 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.713+0000 7ff2d6ffd700 1 -- 172.21.15.195:0/203624633 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc005040 con 0x7ff2d80f6a10 2023-10-12T22:37:49.718 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.713+0000 7ff2d6ffd700 1 --2- 172.21.15.195:0/203624633 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2d80f6a10 0x7ff2d8106700 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ff2cc013510 tx=0x7ff2cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:49.718 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.713+0000 7ff2bf7fe700 1 -- 172.21.15.195:0/203624633 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2cc009320 con 0x7ff2d80f6a10 2023-10-12T22:37:49.718 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.713+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/203624633 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d8106df0 con 0x7ff2d80f6a10 2023-10-12T22:37:49.718 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.713+0000 7ff2bf7fe700 1 -- 172.21.15.195:0/203624633 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2cc013f00 con 0x7ff2d80f6a10 2023-10-12T22:37:49.719 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.713+0000 7ff2bf7fe700 1 -- 172.21.15.195:0/203624633 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2cc014d90 con 0x7ff2d80f6a10 2023-10-12T22:37:49.719 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.713+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/203624633 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d8107280 con 0x7ff2d80f6a10 2023-10-12T22:37:49.719 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.714+0000 7ff2bf7fe700 1 -- 172.21.15.195:0/203624633 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2cc0095f0 con 0x7ff2d80f6a10 2023-10-12T22:37:49.719 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.714+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/203624633 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d8045ad0 con 0x7ff2d80f6a10 2023-10-12T22:37:49.720 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.714+0000 7ff2bf7fe700 1 --2- 172.21.15.195:0/203624633 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff2c002ff60 0x7ff2c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:49.720 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.714+0000 7ff2bf7fe700 1 -- 172.21.15.195:0/203624633 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2cc040fe0 con 0x7ff2d80f6a10 2023-10-12T22:37:49.720 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.714+0000 7ff2d67fc700 1 --2- 172.21.15.195:0/203624633 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff2c002ff60 0x7ff2c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:49.721 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.715+0000 7ff2d67fc700 1 --2- 172.21.15.195:0/203624633 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff2c002ff60 0x7ff2c0032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ff2c80096a0 tx=0x7ff2c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:49.721 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.719+0000 7ff2bf7fe700 1 -- 172.21.15.195:0/203624633 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2cc014420 con 0x7ff2d80f6a10 2023-10-12T22:37:49.926 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.925+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/203624633 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2d81080d0 con 0x7ff2d80f6a10 2023-10-12T22:37:49.927 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.925+0000 7ff2bf7fe700 1 -- 172.21.15.195:0/203624633 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff2cc010230 con 0x7ff2d80f6a10 2023-10-12T22:37:49.927 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:49.927 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:49.928 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.927+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/203624633 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff2c002ff60 msgr2=0x7ff2c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:49.928 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.927+0000 7ff2dd7c7700 1 --2- 172.21.15.195:0/203624633 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff2c002ff60 0x7ff2c0032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ff2c80096a0 tx=0x7ff2c8006b40).stop 2023-10-12T22:37:49.929 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.927+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/203624633 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2d80f6a10 msgr2=0x7ff2d8106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:49.929 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.927+0000 7ff2dd7c7700 1 --2- 172.21.15.195:0/203624633 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2d80f6a10 0x7ff2d8106700 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ff2cc013510 tx=0x7ff2cc00b040).stop 2023-10-12T22:37:49.929 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.927+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/203624633 shutdown_connections 2023-10-12T22:37:49.929 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.927+0000 7ff2dd7c7700 1 --2- 172.21.15.195:0/203624633 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff2c002ff60 0x7ff2c0032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:49.929 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.927+0000 7ff2dd7c7700 1 --2- 172.21.15.195:0/203624633 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2d80f6a10 0x7ff2d8106700 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:49.930 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.927+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/203624633 >> 172.21.15.195:0/203624633 conn(0x7ff2d80f02e0 msgr2=0x7ff2d80f0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:49.930 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.927+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/203624633 shutdown_connections 2023-10-12T22:37:49.930 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:49.927+0000 7ff2dd7c7700 1 -- 172.21.15.195:0/203624633 wait complete. 2023-10-12T22:37:49.930 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:50.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:48.856920+0000 mgr.smithi131.efktma (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:50.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:49.926823+0000 mon.smithi131 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.195:0/203624633' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:51.355 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:51.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:52.537 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.535+0000 7f55b359e700 1 -- 172.21.15.195:0/852063172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55b40f51a0 msgr2=0x7f55b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:52.537 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.535+0000 7f55b359e700 1 --2- 172.21.15.195:0/852063172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55b40f51a0 0x7f55b40f55c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f55a8009fb0 tx=0x7f55a8009b30).stop 2023-10-12T22:37:52.538 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.535+0000 7f55b359e700 1 -- 172.21.15.195:0/852063172 shutdown_connections 2023-10-12T22:37:52.538 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.535+0000 7f55b359e700 1 --2- 172.21.15.195:0/852063172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55b40f51a0 0x7f55b40f55c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:52.538 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.535+0000 7f55b359e700 1 -- 172.21.15.195:0/852063172 >> 172.21.15.195:0/852063172 conn(0x7f55b40f09a0 msgr2=0x7f55b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:52.538 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.535+0000 7f55b359e700 1 -- 172.21.15.195:0/852063172 shutdown_connections 2023-10-12T22:37:52.538 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.535+0000 7f55b359e700 1 -- 172.21.15.195:0/852063172 wait complete. 2023-10-12T22:37:52.539 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.535+0000 7f55b359e700 1 Processor -- start 2023-10-12T22:37:52.539 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.535+0000 7f55b359e700 1 -- start start 2023-10-12T22:37:52.539 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.536+0000 7f55b359e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55b40f51a0 0x7f55b41086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:52.539 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.536+0000 7f55b359e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b4108bb0 con 0x7f55b40f51a0 2023-10-12T22:37:52.539 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.536+0000 7f55b259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55b40f51a0 0x7f55b41086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:52.540 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.536+0000 7f55b259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55b40f51a0 0x7f55b41086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:37800/0 (socket says 172.21.15.195:37800) 2023-10-12T22:37:52.540 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.536+0000 7f55b259c700 1 -- 172.21.15.195:0/66802869 learned_addr learned my addr 172.21.15.195:0/66802869 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:52.540 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.536+0000 7f55b259c700 1 -- 172.21.15.195:0/66802869 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a8005040 con 0x7f55b40f51a0 2023-10-12T22:37:52.540 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.536+0000 7f55b259c700 1 --2- 172.21.15.195:0/66802869 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55b40f51a0 0x7f55b41086c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f55a8013510 tx=0x7f55a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:52.541 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.536+0000 7f55a2ffd700 1 -- 172.21.15.195:0/66802869 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55a8009320 con 0x7f55b40f51a0 2023-10-12T22:37:52.542 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.536+0000 7f55b359e700 1 -- 172.21.15.195:0/66802869 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b4108db0 con 0x7f55b40f51a0 2023-10-12T22:37:52.542 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.536+0000 7f55a2ffd700 1 -- 172.21.15.195:0/66802869 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55a8013f00 con 0x7f55b40f51a0 2023-10-12T22:37:52.542 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.536+0000 7f55a2ffd700 1 -- 172.21.15.195:0/66802869 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55a8014dc0 con 0x7f55b40f51a0 2023-10-12T22:37:52.542 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.536+0000 7f55b359e700 1 -- 172.21.15.195:0/66802869 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b4109240 con 0x7f55b40f51a0 2023-10-12T22:37:52.543 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.537+0000 7f55a2ffd700 1 -- 172.21.15.195:0/66802869 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55a80095f0 con 0x7f55b40f51a0 2023-10-12T22:37:52.543 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.537+0000 7f55b359e700 1 -- 172.21.15.195:0/66802869 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b4045ad0 con 0x7f55b40f51a0 2023-10-12T22:37:52.543 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.537+0000 7f55a2ffd700 1 --2- 172.21.15.195:0/66802869 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f559c02ffb0 0x7f559c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:52.543 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.537+0000 7f55a2ffd700 1 -- 172.21.15.195:0/66802869 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55a80410c0 con 0x7f55b40f51a0 2023-10-12T22:37:52.543 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.538+0000 7f55b1d9b700 1 --2- 172.21.15.195:0/66802869 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f559c02ffb0 0x7f559c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:52.543 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.538+0000 7f55b1d9b700 1 --2- 172.21.15.195:0/66802869 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f559c02ffb0 0x7f559c032470 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f55a40096a0 tx=0x7f55a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:52.544 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.542+0000 7f55a2ffd700 1 -- 172.21.15.195:0/66802869 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55a801c790 con 0x7f55b40f51a0 2023-10-12T22:37:52.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.748+0000 7f55b359e700 1 -- 172.21.15.195:0/66802869 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55b410a0f0 con 0x7f55b40f51a0 2023-10-12T22:37:52.751 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.749+0000 7f55a2ffd700 1 -- 172.21.15.195:0/66802869 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f55a8010230 con 0x7f55b40f51a0 2023-10-12T22:37:52.751 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:52.752 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:52.752 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.751+0000 7f55b359e700 1 -- 172.21.15.195:0/66802869 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f559c02ffb0 msgr2=0x7f559c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:52.753 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.751+0000 7f55b359e700 1 --2- 172.21.15.195:0/66802869 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f559c02ffb0 0x7f559c032470 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f55a40096a0 tx=0x7f55a4006b40).stop 2023-10-12T22:37:52.753 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.751+0000 7f55b359e700 1 -- 172.21.15.195:0/66802869 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55b40f51a0 msgr2=0x7f55b41086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:52.753 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.751+0000 7f55b359e700 1 --2- 172.21.15.195:0/66802869 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55b40f51a0 0x7f55b41086c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f55a8013510 tx=0x7f55a800b040).stop 2023-10-12T22:37:52.753 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.751+0000 7f55b359e700 1 -- 172.21.15.195:0/66802869 shutdown_connections 2023-10-12T22:37:52.754 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.751+0000 7f55b359e700 1 --2- 172.21.15.195:0/66802869 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f559c02ffb0 0x7f559c032470 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:52.754 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.751+0000 7f55b359e700 1 --2- 172.21.15.195:0/66802869 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55b40f51a0 0x7f55b41086c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:52.754 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.751+0000 7f55b359e700 1 -- 172.21.15.195:0/66802869 >> 172.21.15.195:0/66802869 conn(0x7f55b40f09a0 msgr2=0x7f55b405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:52.754 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.751+0000 7f55b359e700 1 -- 172.21.15.195:0/66802869 shutdown_connections 2023-10-12T22:37:52.754 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:52.751+0000 7f55b359e700 1 -- 172.21.15.195:0/66802869 wait complete. 2023-10-12T22:37:52.755 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:52.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:50.857226+0000 mgr.smithi131.efktma (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:53.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:53 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:52.750458+0000 mon.smithi131 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.195:0/66802869' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:54.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:54.144 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:54.989 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:52.857545+0000 mgr.smithi131.efktma (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:55.318 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.317+0000 7facc10c3700 1 -- 172.21.15.195:0/3032073808 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7facbc0f51a0 msgr2=0x7facbc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:55.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.317+0000 7facc10c3700 1 --2- 172.21.15.195:0/3032073808 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7facbc0f51a0 0x7facbc0f55c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7facac009fb0 tx=0x7facac009b30).stop 2023-10-12T22:37:55.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.317+0000 7facc10c3700 1 -- 172.21.15.195:0/3032073808 shutdown_connections 2023-10-12T22:37:55.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.317+0000 7facc10c3700 1 --2- 172.21.15.195:0/3032073808 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7facbc0f51a0 0x7facbc0f55c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:55.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.317+0000 7facc10c3700 1 -- 172.21.15.195:0/3032073808 >> 172.21.15.195:0/3032073808 conn(0x7facbc0f09a0 msgr2=0x7facbc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:55.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.317+0000 7facc10c3700 1 -- 172.21.15.195:0/3032073808 shutdown_connections 2023-10-12T22:37:55.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.317+0000 7facc10c3700 1 -- 172.21.15.195:0/3032073808 wait complete. 2023-10-12T22:37:55.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.317+0000 7facc10c3700 1 Processor -- start 2023-10-12T22:37:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.318+0000 7facc10c3700 1 -- start start 2023-10-12T22:37:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.318+0000 7facc10c3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7facbc0f51a0 0x7facbc1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.318+0000 7facc10c3700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc1058a0 con 0x7facbc0f51a0 2023-10-12T22:37:55.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.318+0000 7facbbfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7facbc0f51a0 0x7facbc1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:55.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.318+0000 7facbbfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7facbc0f51a0 0x7facbc1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:37814/0 (socket says 172.21.15.195:37814) 2023-10-12T22:37:55.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.318+0000 7facbbfff700 1 -- 172.21.15.195:0/2221543512 learned_addr learned my addr 172.21.15.195:0/2221543512 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:55.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.318+0000 7facbbfff700 1 -- 172.21.15.195:0/2221543512 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac005040 con 0x7facbc0f51a0 2023-10-12T22:37:55.323 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.319+0000 7facbbfff700 1 --2- 172.21.15.195:0/2221543512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7facbc0f51a0 0x7facbc1053b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7facac013510 tx=0x7facac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:55.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.319+0000 7facb8ff9700 1 -- 172.21.15.195:0/2221543512 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facac009320 con 0x7facbc0f51a0 2023-10-12T22:37:55.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.319+0000 7facc10c3700 1 -- 172.21.15.195:0/2221543512 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facbc105aa0 con 0x7facbc0f51a0 2023-10-12T22:37:55.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.319+0000 7facb8ff9700 1 -- 172.21.15.195:0/2221543512 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facac013f00 con 0x7facbc0f51a0 2023-10-12T22:37:55.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.319+0000 7facb8ff9700 1 -- 172.21.15.195:0/2221543512 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facac014dc0 con 0x7facbc0f51a0 2023-10-12T22:37:55.324 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.319+0000 7facc10c3700 1 -- 172.21.15.195:0/2221543512 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc105f30 con 0x7facbc0f51a0 2023-10-12T22:37:55.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.320+0000 7facb8ff9700 1 -- 172.21.15.195:0/2221543512 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7facac0095f0 con 0x7facbc0f51a0 2023-10-12T22:37:55.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.320+0000 7facc10c3700 1 -- 172.21.15.195:0/2221543512 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facbc045ad0 con 0x7facbc0f51a0 2023-10-12T22:37:55.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.320+0000 7facb8ff9700 1 --2- 172.21.15.195:0/2221543512 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7faca402ff60 0x7faca4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:55.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.320+0000 7facb8ff9700 1 -- 172.21.15.195:0/2221543512 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7facac041000 con 0x7facbc0f51a0 2023-10-12T22:37:55.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.320+0000 7facbb7fe700 1 --2- 172.21.15.195:0/2221543512 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7faca402ff60 0x7faca4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:55.326 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.320+0000 7facbb7fe700 1 --2- 172.21.15.195:0/2221543512 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7faca402ff60 0x7faca4032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7facb00096a0 tx=0x7facb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:55.326 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.324+0000 7facb8ff9700 1 -- 172.21.15.195:0/2221543512 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7facac014420 con 0x7facbc0f51a0 2023-10-12T22:37:55.532 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.531+0000 7facc10c3700 1 -- 172.21.15.195:0/2221543512 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7facbc106de0 con 0x7facbc0f51a0 2023-10-12T22:37:55.533 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.532+0000 7facb8ff9700 1 -- 172.21.15.195:0/2221543512 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7facac010230 con 0x7facbc0f51a0 2023-10-12T22:37:55.533 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:55.534 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.533+0000 7facc10c3700 1 -- 172.21.15.195:0/2221543512 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7faca402ff60 msgr2=0x7faca4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.533+0000 7facc10c3700 1 --2- 172.21.15.195:0/2221543512 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7faca402ff60 0x7faca4032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7facb00096a0 tx=0x7facb0006b40).stop 2023-10-12T22:37:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.533+0000 7facc10c3700 1 -- 172.21.15.195:0/2221543512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7facbc0f51a0 msgr2=0x7facbc1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.533+0000 7facc10c3700 1 --2- 172.21.15.195:0/2221543512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7facbc0f51a0 0x7facbc1053b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7facac013510 tx=0x7facac00b040).stop 2023-10-12T22:37:55.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.533+0000 7facc10c3700 1 -- 172.21.15.195:0/2221543512 shutdown_connections 2023-10-12T22:37:55.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.533+0000 7facc10c3700 1 --2- 172.21.15.195:0/2221543512 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7faca402ff60 0x7faca4032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:55.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.533+0000 7facc10c3700 1 --2- 172.21.15.195:0/2221543512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7facbc0f51a0 0x7facbc1053b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:55.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.533+0000 7facc10c3700 1 -- 172.21.15.195:0/2221543512 >> 172.21.15.195:0/2221543512 conn(0x7facbc0f09a0 msgr2=0x7facbc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:55.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.534+0000 7facc10c3700 1 -- 172.21.15.195:0/2221543512 shutdown_connections 2023-10-12T22:37:55.536 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:55.534+0000 7facc10c3700 1 -- 172.21.15.195:0/2221543512 wait complete. 2023-10-12T22:37:55.537 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:56.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:55 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:54.857900+0000 mgr.smithi131.efktma (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:56.240 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:55 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:55.533024+0000 mon.smithi131 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.195:0/2221543512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:56.916 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:56.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:37:58.099 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.096+0000 7f1c47298700 1 -- 172.21.15.195:0/273116810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c400f5410 msgr2=0x7f1c400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:58.100 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.096+0000 7f1c47298700 1 --2- 172.21.15.195:0/273116810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c400f5410 0x7f1c400f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f1c3c009fb0 tx=0x7f1c3c009b30).stop 2023-10-12T22:37:58.100 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.096+0000 7f1c47298700 1 -- 172.21.15.195:0/273116810 shutdown_connections 2023-10-12T22:37:58.100 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.096+0000 7f1c47298700 1 --2- 172.21.15.195:0/273116810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c400f5410 0x7f1c400f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:58.101 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.096+0000 7f1c47298700 1 -- 172.21.15.195:0/273116810 >> 172.21.15.195:0/273116810 conn(0x7f1c400f0b70 msgr2=0x7f1c400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:58.101 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.096+0000 7f1c47298700 1 -- 172.21.15.195:0/273116810 shutdown_connections 2023-10-12T22:37:58.101 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.096+0000 7f1c47298700 1 -- 172.21.15.195:0/273116810 wait complete. 2023-10-12T22:37:58.101 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.096+0000 7f1c47298700 1 Processor -- start 2023-10-12T22:37:58.102 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.097+0000 7f1c47298700 1 -- start start 2023-10-12T22:37:58.102 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.097+0000 7f1c47298700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c400f5410 0x7f1c40105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:58.102 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.097+0000 7f1c47298700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c40105af0 con 0x7f1c400f5410 2023-10-12T22:37:58.102 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.097+0000 7f1c45034700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c400f5410 0x7f1c40105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:58.102 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.097+0000 7f1c45034700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c400f5410 0x7f1c40105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:37828/0 (socket says 172.21.15.195:37828) 2023-10-12T22:37:58.103 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.097+0000 7f1c45034700 1 -- 172.21.15.195:0/3090020798 learned_addr learned my addr 172.21.15.195:0/3090020798 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:37:58.103 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.097+0000 7f1c45034700 1 -- 172.21.15.195:0/3090020798 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c005040 con 0x7f1c400f5410 2023-10-12T22:37:58.103 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.097+0000 7f1c45034700 1 --2- 172.21.15.195:0/3090020798 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c400f5410 0x7f1c40105600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f1c3c013510 tx=0x7f1c3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:58.103 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.098+0000 7f1c35ffb700 1 -- 172.21.15.195:0/3090020798 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c3c009320 con 0x7f1c400f5410 2023-10-12T22:37:58.103 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.098+0000 7f1c47298700 1 -- 172.21.15.195:0/3090020798 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c40105cf0 con 0x7f1c400f5410 2023-10-12T22:37:58.104 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.098+0000 7f1c35ffb700 1 -- 172.21.15.195:0/3090020798 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c3c013f00 con 0x7f1c400f5410 2023-10-12T22:37:58.104 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.098+0000 7f1c35ffb700 1 -- 172.21.15.195:0/3090020798 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c3c014d90 con 0x7f1c400f5410 2023-10-12T22:37:58.104 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.098+0000 7f1c47298700 1 -- 172.21.15.195:0/3090020798 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c40106180 con 0x7f1c400f5410 2023-10-12T22:37:58.105 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.099+0000 7f1c35ffb700 1 -- 172.21.15.195:0/3090020798 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c3c0095f0 con 0x7f1c400f5410 2023-10-12T22:37:58.105 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.099+0000 7f1c47298700 1 -- 172.21.15.195:0/3090020798 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c40045ad0 con 0x7f1c400f5410 2023-10-12T22:37:58.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.099+0000 7f1c35ffb700 1 --2- 172.21.15.195:0/3090020798 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1c2c02ff60 0x7f1c2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:37:58.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.099+0000 7f1c35ffb700 1 -- 172.21.15.195:0/3090020798 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c3c042dc0 con 0x7f1c400f5410 2023-10-12T22:37:58.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.099+0000 7f1c44833700 1 --2- 172.21.15.195:0/3090020798 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1c2c02ff60 0x7f1c2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:37:58.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.099+0000 7f1c44833700 1 --2- 172.21.15.195:0/3090020798 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1c2c02ff60 0x7f1c2c032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f1c300096a0 tx=0x7f1c30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:37:58.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.104+0000 7f1c35ffb700 1 -- 172.21.15.195:0/3090020798 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c3c014420 con 0x7f1c400f5410 2023-10-12T22:37:58.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:57 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:56.858259+0000 mgr.smithi131.efktma (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:37:58.312 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.311+0000 7f1c47298700 1 -- 172.21.15.195:0/3090020798 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c400fae60 con 0x7f1c400f5410 2023-10-12T22:37:58.313 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.311+0000 7f1c35ffb700 1 -- 172.21.15.195:0/3090020798 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1c3c011290 con 0x7f1c400f5410 2023-10-12T22:37:58.313 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:37:58.313 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:37:58.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.313+0000 7f1c47298700 1 -- 172.21.15.195:0/3090020798 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1c2c02ff60 msgr2=0x7f1c2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:58.315 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.313+0000 7f1c47298700 1 --2- 172.21.15.195:0/3090020798 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1c2c02ff60 0x7f1c2c032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f1c300096a0 tx=0x7f1c30006b40).stop 2023-10-12T22:37:58.315 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.313+0000 7f1c47298700 1 -- 172.21.15.195:0/3090020798 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c400f5410 msgr2=0x7f1c40105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:37:58.315 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.313+0000 7f1c47298700 1 --2- 172.21.15.195:0/3090020798 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c400f5410 0x7f1c40105600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f1c3c013510 tx=0x7f1c3c00b040).stop 2023-10-12T22:37:58.315 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.313+0000 7f1c47298700 1 -- 172.21.15.195:0/3090020798 shutdown_connections 2023-10-12T22:37:58.315 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.313+0000 7f1c47298700 1 --2- 172.21.15.195:0/3090020798 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1c2c02ff60 0x7f1c2c032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:58.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.313+0000 7f1c47298700 1 --2- 172.21.15.195:0/3090020798 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c400f5410 0x7f1c40105600 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:37:58.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.313+0000 7f1c47298700 1 -- 172.21.15.195:0/3090020798 >> 172.21.15.195:0/3090020798 conn(0x7f1c400f0b70 msgr2=0x7f1c400f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:37:58.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.313+0000 7f1c47298700 1 -- 172.21.15.195:0/3090020798 shutdown_connections 2023-10-12T22:37:58.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:37:58.313+0000 7f1c47298700 1 -- 172.21.15.195:0/3090020798 wait complete. 2023-10-12T22:37:58.317 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:37:59.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:58 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:37:58.312900+0000 mon.smithi131 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.195:0/3090020798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:37:59.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:37:59.694 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:00.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:37:59 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:37:58.858634+0000 mgr.smithi131.efktma (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:00.836 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.835+0000 7fb7e8347700 1 -- 172.21.15.195:0/3954047810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7e00f6510 msgr2=0x7fb7e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:00.837 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.835+0000 7fb7e8347700 1 --2- 172.21.15.195:0/3954047810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7e00f6510 0x7fb7e00f6930 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fb7d0009fb0 tx=0x7fb7d0009b30).stop 2023-10-12T22:38:00.837 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.835+0000 7fb7e8347700 1 -- 172.21.15.195:0/3954047810 shutdown_connections 2023-10-12T22:38:00.837 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.835+0000 7fb7e8347700 1 --2- 172.21.15.195:0/3954047810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7e00f6510 0x7fb7e00f6930 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:00.837 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.835+0000 7fb7e8347700 1 -- 172.21.15.195:0/3954047810 >> 172.21.15.195:0/3954047810 conn(0x7fb7e00f1c70 msgr2=0x7fb7e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:00.837 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.835+0000 7fb7e8347700 1 -- 172.21.15.195:0/3954047810 shutdown_connections 2023-10-12T22:38:00.838 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.835+0000 7fb7e8347700 1 -- 172.21.15.195:0/3954047810 wait complete. 2023-10-12T22:38:00.838 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.835+0000 7fb7e8347700 1 Processor -- start 2023-10-12T22:38:00.838 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.835+0000 7fb7e8347700 1 -- start start 2023-10-12T22:38:00.840 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.835+0000 7fb7e8347700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7e00f6510 0x7fb7e0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:00.840 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.835+0000 7fb7e8347700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e0106c20 con 0x7fb7e00f6510 2023-10-12T22:38:00.840 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.836+0000 7fb7e60e3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7e00f6510 0x7fb7e0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:00.840 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.836+0000 7fb7e60e3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7e00f6510 0x7fb7e0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38708/0 (socket says 172.21.15.195:38708) 2023-10-12T22:38:00.841 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.836+0000 7fb7e60e3700 1 -- 172.21.15.195:0/392559075 learned_addr learned my addr 172.21.15.195:0/392559075 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:00.841 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.836+0000 7fb7e60e3700 1 -- 172.21.15.195:0/392559075 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d0005040 con 0x7fb7e00f6510 2023-10-12T22:38:00.841 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.836+0000 7fb7e60e3700 1 --2- 172.21.15.195:0/392559075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7e00f6510 0x7fb7e0106730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fb7d0013510 tx=0x7fb7d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:00.841 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.836+0000 7fb7d6ffd700 1 -- 172.21.15.195:0/392559075 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7d0009320 con 0x7fb7e00f6510 2023-10-12T22:38:00.841 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.836+0000 7fb7e8347700 1 -- 172.21.15.195:0/392559075 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e0106e20 con 0x7fb7e00f6510 2023-10-12T22:38:00.841 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.836+0000 7fb7d6ffd700 1 -- 172.21.15.195:0/392559075 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7d0013f00 con 0x7fb7e00f6510 2023-10-12T22:38:00.842 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.836+0000 7fb7d6ffd700 1 -- 172.21.15.195:0/392559075 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7d0014d90 con 0x7fb7e00f6510 2023-10-12T22:38:00.842 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.836+0000 7fb7e8347700 1 -- 172.21.15.195:0/392559075 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e01072b0 con 0x7fb7e00f6510 2023-10-12T22:38:00.842 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.837+0000 7fb7d6ffd700 1 -- 172.21.15.195:0/392559075 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7d00095f0 con 0x7fb7e00f6510 2023-10-12T22:38:00.842 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.837+0000 7fb7d6ffd700 1 --2- 172.21.15.195:0/392559075 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb7cc02ff60 0x7fb7cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:00.842 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.837+0000 7fb7e8347700 1 -- 172.21.15.195:0/392559075 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e0045ad0 con 0x7fb7e00f6510 2023-10-12T22:38:00.843 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.837+0000 7fb7d6ffd700 1 -- 172.21.15.195:0/392559075 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb7d0040fe0 con 0x7fb7e00f6510 2023-10-12T22:38:00.843 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.837+0000 7fb7e58e2700 1 --2- 172.21.15.195:0/392559075 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb7cc02ff60 0x7fb7cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:00.843 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.838+0000 7fb7e58e2700 1 --2- 172.21.15.195:0/392559075 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb7cc02ff60 0x7fb7cc032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb7dc0096a0 tx=0x7fb7dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:00.843 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:00.842+0000 7fb7d6ffd700 1 -- 172.21.15.195:0/392559075 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7d001d7f0 con 0x7fb7e00f6510 2023-10-12T22:38:01.052 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:01.047+0000 7fb7e8347700 1 -- 172.21.15.195:0/392559075 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7e00fc050 con 0x7fb7e00f6510 2023-10-12T22:38:01.053 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:01.048+0000 7fb7d6ffd700 1 -- 172.21.15.195:0/392559075 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb7d003e090 con 0x7fb7e00f6510 2023-10-12T22:38:01.053 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:01.053 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:01.054 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:01.049+0000 7fb7e8347700 1 -- 172.21.15.195:0/392559075 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb7cc02ff60 msgr2=0x7fb7cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:01.055 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:01.049+0000 7fb7e8347700 1 --2- 172.21.15.195:0/392559075 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb7cc02ff60 0x7fb7cc032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb7dc0096a0 tx=0x7fb7dc006b40).stop 2023-10-12T22:38:01.055 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:01.049+0000 7fb7e8347700 1 -- 172.21.15.195:0/392559075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7e00f6510 msgr2=0x7fb7e0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:01.055 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:01.049+0000 7fb7e8347700 1 --2- 172.21.15.195:0/392559075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7e00f6510 0x7fb7e0106730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fb7d0013510 tx=0x7fb7d000b040).stop 2023-10-12T22:38:01.055 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:01.049+0000 7fb7e8347700 1 -- 172.21.15.195:0/392559075 shutdown_connections 2023-10-12T22:38:01.055 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:01.049+0000 7fb7e8347700 1 --2- 172.21.15.195:0/392559075 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fb7cc02ff60 0x7fb7cc032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:01.056 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:01.049+0000 7fb7e8347700 1 --2- 172.21.15.195:0/392559075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7e00f6510 0x7fb7e0106730 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:01.056 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:01.049+0000 7fb7e8347700 1 -- 172.21.15.195:0/392559075 >> 172.21.15.195:0/392559075 conn(0x7fb7e00f1c70 msgr2=0x7fb7e00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:01.056 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:01.049+0000 7fb7e8347700 1 -- 172.21.15.195:0/392559075 shutdown_connections 2023-10-12T22:38:01.056 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:01.049+0000 7fb7e8347700 1 -- 172.21.15.195:0/392559075 wait complete. 2023-10-12T22:38:01.056 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:02.417 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:02.418 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:02.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:01 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:00.858952+0000 mgr.smithi131.efktma (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:02.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:01 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:01.048906+0000 mon.smithi131 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.195:0/392559075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:03.570 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.567+0000 7f572957e700 1 -- 172.21.15.195:0/2001517224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57240f5410 msgr2=0x7f57240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:03.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.567+0000 7f572957e700 1 --2- 172.21.15.195:0/2001517224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57240f5410 0x7f57240f5830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f5718009fb0 tx=0x7f5718009b30).stop 2023-10-12T22:38:03.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.567+0000 7f572957e700 1 -- 172.21.15.195:0/2001517224 shutdown_connections 2023-10-12T22:38:03.571 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.567+0000 7f572957e700 1 --2- 172.21.15.195:0/2001517224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57240f5410 0x7f57240f5830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.567+0000 7f572957e700 1 -- 172.21.15.195:0/2001517224 >> 172.21.15.195:0/2001517224 conn(0x7f57240f0b70 msgr2=0x7f57240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.567+0000 7f572957e700 1 -- 172.21.15.195:0/2001517224 shutdown_connections 2023-10-12T22:38:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.567+0000 7f572957e700 1 -- 172.21.15.195:0/2001517224 wait complete. 2023-10-12T22:38:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.568+0000 7f572957e700 1 Processor -- start 2023-10-12T22:38:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.568+0000 7f572957e700 1 -- start start 2023-10-12T22:38:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.568+0000 7f572957e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57240f5410 0x7f572410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.568+0000 7f572957e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5724107240 con 0x7f57240f5410 2023-10-12T22:38:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.568+0000 7f5722ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57240f5410 0x7f572410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.568+0000 7f5722ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57240f5410 0x7f572410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38722/0 (socket says 172.21.15.195:38722) 2023-10-12T22:38:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.568+0000 7f5722ffd700 1 -- 172.21.15.195:0/1401474732 learned_addr learned my addr 172.21.15.195:0/1401474732 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.569+0000 7f5722ffd700 1 -- 172.21.15.195:0/1401474732 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5718005040 con 0x7f57240f5410 2023-10-12T22:38:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.569+0000 7f5722ffd700 1 --2- 172.21.15.195:0/1401474732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57240f5410 0x7f572410b850 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f5718013510 tx=0x7f571800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.569+0000 7f570b7fe700 1 -- 172.21.15.195:0/1401474732 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5718009320 con 0x7f57240f5410 2023-10-12T22:38:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.569+0000 7f572957e700 1 -- 172.21.15.195:0/1401474732 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5724060220 con 0x7f57240f5410 2023-10-12T22:38:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.569+0000 7f570b7fe700 1 -- 172.21.15.195:0/1401474732 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5718013f00 con 0x7f57240f5410 2023-10-12T22:38:03.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.569+0000 7f570b7fe700 1 -- 172.21.15.195:0/1401474732 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5718014d90 con 0x7f57240f5410 2023-10-12T22:38:03.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.569+0000 7f572957e700 1 -- 172.21.15.195:0/1401474732 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572410bd40 con 0x7f57240f5410 2023-10-12T22:38:03.575 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.570+0000 7f570b7fe700 1 -- 172.21.15.195:0/1401474732 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5718014420 con 0x7f57240f5410 2023-10-12T22:38:03.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.570+0000 7f572957e700 1 -- 172.21.15.195:0/1401474732 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5724045ad0 con 0x7f57240f5410 2023-10-12T22:38:03.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.570+0000 7f570b7fe700 1 --2- 172.21.15.195:0/1401474732 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f570c02ff60 0x7f570c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:03.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.570+0000 7f570b7fe700 1 -- 172.21.15.195:0/1401474732 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5718041050 con 0x7f57240f5410 2023-10-12T22:38:03.576 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.570+0000 7f57227fc700 1 --2- 172.21.15.195:0/1401474732 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f570c02ff60 0x7f570c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:03.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.571+0000 7f57227fc700 1 --2- 172.21.15.195:0/1401474732 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f570c02ff60 0x7f570c032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f57140096a0 tx=0x7f5714006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:03.577 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.575+0000 7f570b7fe700 1 -- 172.21.15.195:0/1401474732 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57180095f0 con 0x7f57240f5410 2023-10-12T22:38:03.781 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.780+0000 7f572957e700 1 -- 172.21.15.195:0/1401474732 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57240fae60 con 0x7f57240f5410 2023-10-12T22:38:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.781+0000 7f570b7fe700 1 -- 172.21.15.195:0/1401474732 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f571801db70 con 0x7f57240f5410 2023-10-12T22:38:03.782 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:03.783 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:03.784 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.783+0000 7f572957e700 1 -- 172.21.15.195:0/1401474732 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f570c02ff60 msgr2=0x7f570c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:03.784 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.783+0000 7f572957e700 1 --2- 172.21.15.195:0/1401474732 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f570c02ff60 0x7f570c032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f57140096a0 tx=0x7f5714006b40).stop 2023-10-12T22:38:03.784 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.783+0000 7f572957e700 1 -- 172.21.15.195:0/1401474732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57240f5410 msgr2=0x7f572410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:03.784 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.783+0000 7f572957e700 1 --2- 172.21.15.195:0/1401474732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57240f5410 0x7f572410b850 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f5718013510 tx=0x7f571800b040).stop 2023-10-12T22:38:03.785 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.783+0000 7f572957e700 1 -- 172.21.15.195:0/1401474732 shutdown_connections 2023-10-12T22:38:03.785 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.783+0000 7f572957e700 1 --2- 172.21.15.195:0/1401474732 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f570c02ff60 0x7f570c032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:03.785 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.783+0000 7f572957e700 1 --2- 172.21.15.195:0/1401474732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57240f5410 0x7f572410b850 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:03.785 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.783+0000 7f572957e700 1 -- 172.21.15.195:0/1401474732 >> 172.21.15.195:0/1401474732 conn(0x7f57240f0b70 msgr2=0x7f57240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:03.785 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.783+0000 7f572957e700 1 -- 172.21.15.195:0/1401474732 shutdown_connections 2023-10-12T22:38:03.786 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:03.783+0000 7f572957e700 1 -- 172.21.15.195:0/1401474732 wait complete. 2023-10-12T22:38:03.786 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:04.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:03 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:02.859322+0000 mgr.smithi131.efktma (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:04.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:03 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:03.782488+0000 mon.smithi131 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.195:0/1401474732' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:05.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:05.163 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:06.313 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.312+0000 7ff25e963700 1 -- 172.21.15.195:0/1283868094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff258060200 msgr2=0x7ff2580fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:06.313 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.312+0000 7ff25e963700 1 --2- 172.21.15.195:0/1283868094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff258060200 0x7ff2580fb570 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7ff248009fb0 tx=0x7ff248009b30).stop 2023-10-12T22:38:06.313 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.312+0000 7ff25e963700 1 -- 172.21.15.195:0/1283868094 shutdown_connections 2023-10-12T22:38:06.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.312+0000 7ff25e963700 1 --2- 172.21.15.195:0/1283868094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff258060200 0x7ff2580fb570 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:06.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.312+0000 7ff25e963700 1 -- 172.21.15.195:0/1283868094 >> 172.21.15.195:0/1283868094 conn(0x7ff2580f0b70 msgr2=0x7ff2580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:06.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.312+0000 7ff25e963700 1 -- 172.21.15.195:0/1283868094 shutdown_connections 2023-10-12T22:38:06.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.313+0000 7ff25e963700 1 -- 172.21.15.195:0/1283868094 wait complete. 2023-10-12T22:38:06.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.313+0000 7ff25e963700 1 Processor -- start 2023-10-12T22:38:06.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.313+0000 7ff25e963700 1 -- start start 2023-10-12T22:38:06.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.313+0000 7ff25e963700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff258060200 0x7ff2581088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:06.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.313+0000 7ff25e963700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff258108dc0 con 0x7ff258060200 2023-10-12T22:38:06.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.314+0000 7ff257fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff258060200 0x7ff2581088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:06.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.314+0000 7ff257fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff258060200 0x7ff2581088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:38740/0 (socket says 172.21.15.195:38740) 2023-10-12T22:38:06.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.314+0000 7ff257fff700 1 -- 172.21.15.195:0/3245136981 learned_addr learned my addr 172.21.15.195:0/3245136981 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:06.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.314+0000 7ff257fff700 1 -- 172.21.15.195:0/3245136981 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff248005040 con 0x7ff258060200 2023-10-12T22:38:06.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.314+0000 7ff257fff700 1 --2- 172.21.15.195:0/3245136981 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff258060200 0x7ff2581088d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7ff248013510 tx=0x7ff24800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:06.318 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.314+0000 7ff254ff9700 1 -- 172.21.15.195:0/3245136981 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff248009320 con 0x7ff258060200 2023-10-12T22:38:06.318 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.314+0000 7ff25e963700 1 -- 172.21.15.195:0/3245136981 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff258108fc0 con 0x7ff258060200 2023-10-12T22:38:06.318 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.314+0000 7ff254ff9700 1 -- 172.21.15.195:0/3245136981 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff248013f00 con 0x7ff258060200 2023-10-12T22:38:06.318 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.314+0000 7ff254ff9700 1 -- 172.21.15.195:0/3245136981 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff248014dc0 con 0x7ff258060200 2023-10-12T22:38:06.318 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.314+0000 7ff25e963700 1 -- 172.21.15.195:0/3245136981 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff258109450 con 0x7ff258060200 2023-10-12T22:38:06.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.315+0000 7ff254ff9700 1 -- 172.21.15.195:0/3245136981 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2480263a0 con 0x7ff258060200 2023-10-12T22:38:06.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.315+0000 7ff25e963700 1 -- 172.21.15.195:0/3245136981 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff258045ad0 con 0x7ff258060200 2023-10-12T22:38:06.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.315+0000 7ff254ff9700 1 --2- 172.21.15.195:0/3245136981 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff24002ffb0 0x7ff240032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:06.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.315+0000 7ff254ff9700 1 -- 172.21.15.195:0/3245136981 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff248041330 con 0x7ff258060200 2023-10-12T22:38:06.319 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.315+0000 7ff2577fe700 1 --2- 172.21.15.195:0/3245136981 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff24002ffb0 0x7ff240032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:06.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.316+0000 7ff2577fe700 1 --2- 172.21.15.195:0/3245136981 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff24002ffb0 0x7ff240032470 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff24c0096a0 tx=0x7ff24c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:06.320 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.320+0000 7ff254ff9700 1 -- 172.21.15.195:0/3245136981 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff24801d380 con 0x7ff258060200 2023-10-12T22:38:06.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:05 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:04.859690+0000 mgr.smithi131.efktma (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:06.527 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.526+0000 7ff25e963700 1 -- 172.21.15.195:0/3245136981 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff25810a300 con 0x7ff258060200 2023-10-12T22:38:06.527 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.527+0000 7ff254ff9700 1 -- 172.21.15.195:0/3245136981 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff24801cb10 con 0x7ff258060200 2023-10-12T22:38:06.528 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:06.528 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:06.529 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.529+0000 7ff25e963700 1 -- 172.21.15.195:0/3245136981 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff24002ffb0 msgr2=0x7ff240032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:06.530 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.529+0000 7ff25e963700 1 --2- 172.21.15.195:0/3245136981 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff24002ffb0 0x7ff240032470 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff24c0096a0 tx=0x7ff24c006b40).stop 2023-10-12T22:38:06.530 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.529+0000 7ff25e963700 1 -- 172.21.15.195:0/3245136981 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff258060200 msgr2=0x7ff2581088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:06.530 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.529+0000 7ff25e963700 1 --2- 172.21.15.195:0/3245136981 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff258060200 0x7ff2581088d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7ff248013510 tx=0x7ff24800b040).stop 2023-10-12T22:38:06.530 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.529+0000 7ff25e963700 1 -- 172.21.15.195:0/3245136981 shutdown_connections 2023-10-12T22:38:06.530 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.529+0000 7ff25e963700 1 --2- 172.21.15.195:0/3245136981 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff24002ffb0 0x7ff240032470 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:06.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.529+0000 7ff25e963700 1 --2- 172.21.15.195:0/3245136981 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff258060200 0x7ff2581088d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:06.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.529+0000 7ff25e963700 1 -- 172.21.15.195:0/3245136981 >> 172.21.15.195:0/3245136981 conn(0x7ff2580f0b70 msgr2=0x7ff2580f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:06.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.529+0000 7ff25e963700 1 -- 172.21.15.195:0/3245136981 shutdown_connections 2023-10-12T22:38:06.531 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:06.529+0000 7ff25e963700 1 -- 172.21.15.195:0/3245136981 wait complete. 2023-10-12T22:38:06.532 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:07.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:06.527706+0000 mon.smithi131 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.195:0/3245136981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:07.906 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:07.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:08.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:06.860008+0000 mgr.smithi131.efktma (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:09.069 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.067+0000 7fa34842e700 1 -- 172.21.15.195:0/3758978543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa3400f5410 msgr2=0x7fa3400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:09.069 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.067+0000 7fa34842e700 1 --2- 172.21.15.195:0/3758978543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa3400f5410 0x7fa3400f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fa330009fb0 tx=0x7fa330009b30).stop 2023-10-12T22:38:09.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.067+0000 7fa34842e700 1 -- 172.21.15.195:0/3758978543 shutdown_connections 2023-10-12T22:38:09.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.067+0000 7fa34842e700 1 --2- 172.21.15.195:0/3758978543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa3400f5410 0x7fa3400f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:09.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.067+0000 7fa34842e700 1 -- 172.21.15.195:0/3758978543 >> 172.21.15.195:0/3758978543 conn(0x7fa3400f0b70 msgr2=0x7fa3400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:09.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.067+0000 7fa34842e700 1 -- 172.21.15.195:0/3758978543 shutdown_connections 2023-10-12T22:38:09.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.067+0000 7fa34842e700 1 -- 172.21.15.195:0/3758978543 wait complete. 2023-10-12T22:38:09.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.068+0000 7fa34842e700 1 Processor -- start 2023-10-12T22:38:09.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.068+0000 7fa34842e700 1 -- start start 2023-10-12T22:38:09.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.068+0000 7fa34842e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa3400f5410 0x7fa340109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:09.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.068+0000 7fa34842e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa340060220 con 0x7fa3400f5410 2023-10-12T22:38:09.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.068+0000 7fa3461ca700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa3400f5410 0x7fa340109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:09.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.068+0000 7fa3461ca700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa3400f5410 0x7fa340109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:46020/0 (socket says 172.21.15.195:46020) 2023-10-12T22:38:09.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.068+0000 7fa3461ca700 1 -- 172.21.15.195:0/2330686070 learned_addr learned my addr 172.21.15.195:0/2330686070 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:09.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.069+0000 7fa3461ca700 1 -- 172.21.15.195:0/2330686070 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa330005040 con 0x7fa3400f5410 2023-10-12T22:38:09.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.069+0000 7fa3461ca700 1 --2- 172.21.15.195:0/2330686070 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa3400f5410 0x7fa340109450 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fa330013510 tx=0x7fa33000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:09.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.069+0000 7fa336ffd700 1 -- 172.21.15.195:0/2330686070 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa330009320 con 0x7fa3400f5410 2023-10-12T22:38:09.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.069+0000 7fa34842e700 1 -- 172.21.15.195:0/2330686070 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa340060420 con 0x7fa3400f5410 2023-10-12T22:38:09.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.069+0000 7fa336ffd700 1 -- 172.21.15.195:0/2330686070 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa330013f00 con 0x7fa3400f5410 2023-10-12T22:38:09.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.069+0000 7fa336ffd700 1 -- 172.21.15.195:0/2330686070 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa330014d90 con 0x7fa3400f5410 2023-10-12T22:38:09.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.069+0000 7fa34842e700 1 -- 172.21.15.195:0/2330686070 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa340109b00 con 0x7fa3400f5410 2023-10-12T22:38:09.074 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.070+0000 7fa336ffd700 1 -- 172.21.15.195:0/2330686070 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa330014420 con 0x7fa3400f5410 2023-10-12T22:38:09.074 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.070+0000 7fa34842e700 1 -- 172.21.15.195:0/2330686070 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa340059100 con 0x7fa3400f5410 2023-10-12T22:38:09.074 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.070+0000 7fa336ffd700 1 --2- 172.21.15.195:0/2330686070 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa32c02ff60 0x7fa32c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:09.075 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.070+0000 7fa336ffd700 1 -- 172.21.15.195:0/2330686070 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa330041000 con 0x7fa3400f5410 2023-10-12T22:38:09.075 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.070+0000 7fa3459c9700 1 --2- 172.21.15.195:0/2330686070 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa32c02ff60 0x7fa32c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:09.075 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.070+0000 7fa3459c9700 1 --2- 172.21.15.195:0/2330686070 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa32c02ff60 0x7fa32c032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fa33c0096a0 tx=0x7fa33c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:09.075 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.075+0000 7fa336ffd700 1 -- 172.21.15.195:0/2330686070 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3300095f0 con 0x7fa3400f5410 2023-10-12T22:38:09.282 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.282+0000 7fa34842e700 1 -- 172.21.15.195:0/2330686070 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa340045ad0 con 0x7fa3400f5410 2023-10-12T22:38:09.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.282+0000 7fa336ffd700 1 -- 172.21.15.195:0/2330686070 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa33001db70 con 0x7fa3400f5410 2023-10-12T22:38:09.283 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:09.283 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:09.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.284+0000 7fa34842e700 1 -- 172.21.15.195:0/2330686070 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa32c02ff60 msgr2=0x7fa32c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:09.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.284+0000 7fa34842e700 1 --2- 172.21.15.195:0/2330686070 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa32c02ff60 0x7fa32c032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fa33c0096a0 tx=0x7fa33c006b40).stop 2023-10-12T22:38:09.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.284+0000 7fa34842e700 1 -- 172.21.15.195:0/2330686070 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa3400f5410 msgr2=0x7fa340109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:09.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.284+0000 7fa34842e700 1 --2- 172.21.15.195:0/2330686070 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa3400f5410 0x7fa340109450 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fa330013510 tx=0x7fa33000b040).stop 2023-10-12T22:38:09.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.284+0000 7fa34842e700 1 -- 172.21.15.195:0/2330686070 shutdown_connections 2023-10-12T22:38:09.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.284+0000 7fa34842e700 1 --2- 172.21.15.195:0/2330686070 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa32c02ff60 0x7fa32c032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:09.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.284+0000 7fa34842e700 1 --2- 172.21.15.195:0/2330686070 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa3400f5410 0x7fa340109450 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:09.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.284+0000 7fa34842e700 1 -- 172.21.15.195:0/2330686070 >> 172.21.15.195:0/2330686070 conn(0x7fa3400f0b70 msgr2=0x7fa3400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:09.286 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.284+0000 7fa34842e700 1 -- 172.21.15.195:0/2330686070 shutdown_connections 2023-10-12T22:38:09.286 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:09.284+0000 7fa34842e700 1 -- 172.21.15.195:0/2330686070 wait complete. 2023-10-12T22:38:09.286 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:10.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:08.860335+0000 mgr.smithi131.efktma (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:10.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:09.282930+0000 mon.smithi131 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.195:0/2330686070' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:10.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:10.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.893+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3308482065 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcae40f4740 msgr2=0x7fcae40f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.893+0000 7fcaebe3a700 1 --2- 172.21.15.195:0/3308482065 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcae40f4740 0x7fcae40f4b60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fcae0009fb0 tx=0x7fcae0009b30).stop 2023-10-12T22:38:11.895 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.893+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3308482065 shutdown_connections 2023-10-12T22:38:11.895 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.893+0000 7fcaebe3a700 1 --2- 172.21.15.195:0/3308482065 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcae40f4740 0x7fcae40f4b60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:11.895 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.893+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3308482065 >> 172.21.15.195:0/3308482065 conn(0x7fcae40f0260 msgr2=0x7fcae40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:11.895 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.893+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3308482065 shutdown_connections 2023-10-12T22:38:11.895 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.893+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3308482065 wait complete. 2023-10-12T22:38:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.894+0000 7fcaebe3a700 1 Processor -- start 2023-10-12T22:38:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.894+0000 7fcaebe3a700 1 -- start start 2023-10-12T22:38:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.894+0000 7fcaebe3a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcae40f4740 0x7fcae40f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.894+0000 7fcaebe3a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae40f5770 con 0x7fcae40f4740 2023-10-12T22:38:11.898 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.894+0000 7fcae9bd6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcae40f4740 0x7fcae40f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:11.898 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.894+0000 7fcae9bd6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcae40f4740 0x7fcae40f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:46038/0 (socket says 172.21.15.195:46038) 2023-10-12T22:38:11.898 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.894+0000 7fcae9bd6700 1 -- 172.21.15.195:0/3079487046 learned_addr learned my addr 172.21.15.195:0/3079487046 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:11.898 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.895+0000 7fcae9bd6700 1 -- 172.21.15.195:0/3079487046 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae0005040 con 0x7fcae40f4740 2023-10-12T22:38:11.899 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.895+0000 7fcae9bd6700 1 --2- 172.21.15.195:0/3079487046 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcae40f4740 0x7fcae40f72f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fcae0014000 tx=0x7fcae000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:11.899 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.895+0000 7fcada7fc700 1 -- 172.21.15.195:0/3079487046 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcae00095f0 con 0x7fcae40f4740 2023-10-12T22:38:11.899 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.895+0000 7fcada7fc700 1 -- 172.21.15.195:0/3079487046 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcae0007e70 con 0x7fcae40f4740 2023-10-12T22:38:11.899 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.895+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3079487046 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae40f5910 con 0x7fcae40f4740 2023-10-12T22:38:11.899 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.895+0000 7fcada7fc700 1 -- 172.21.15.195:0/3079487046 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcae0014ea0 con 0x7fcae40f4740 2023-10-12T22:38:11.900 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.895+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3079487046 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae40f5da0 con 0x7fcae40f4740 2023-10-12T22:38:11.900 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.896+0000 7fcada7fc700 1 -- 172.21.15.195:0/3079487046 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcae0025070 con 0x7fcae40f4740 2023-10-12T22:38:11.900 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.896+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3079487046 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae4059100 con 0x7fcae40f4740 2023-10-12T22:38:11.900 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.896+0000 7fcada7fc700 1 --2- 172.21.15.195:0/3079487046 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcad002ff60 0x7fcad0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:11.900 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.896+0000 7fcada7fc700 1 -- 172.21.15.195:0/3079487046 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcae0040970 con 0x7fcae40f4740 2023-10-12T22:38:11.900 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.896+0000 7fcae93d5700 1 --2- 172.21.15.195:0/3079487046 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcad002ff60 0x7fcad0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:11.901 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.896+0000 7fcae93d5700 1 --2- 172.21.15.195:0/3079487046 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcad002ff60 0x7fcad0032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcad40096a0 tx=0x7fcad4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:11.901 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:11.900+0000 7fcada7fc700 1 -- 172.21.15.195:0/3079487046 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcae00079c0 con 0x7fcae40f4740 2023-10-12T22:38:12.108 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:12.108+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3079487046 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcae40fbf40 con 0x7fcae40f4740 2023-10-12T22:38:12.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:12.109+0000 7fcada7fc700 1 -- 172.21.15.195:0/3079487046 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcae000f300 con 0x7fcae40f4740 2023-10-12T22:38:12.109 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:12.110 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:12.111 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:12.111+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3079487046 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcad002ff60 msgr2=0x7fcad0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:12.111 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:12.111+0000 7fcaebe3a700 1 --2- 172.21.15.195:0/3079487046 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcad002ff60 0x7fcad0032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcad40096a0 tx=0x7fcad4006b40).stop 2023-10-12T22:38:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:12.111+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3079487046 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcae40f4740 msgr2=0x7fcae40f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:12.111+0000 7fcaebe3a700 1 --2- 172.21.15.195:0/3079487046 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcae40f4740 0x7fcae40f72f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fcae0014000 tx=0x7fcae000b040).stop 2023-10-12T22:38:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:12.111+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3079487046 shutdown_connections 2023-10-12T22:38:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:12.111+0000 7fcaebe3a700 1 --2- 172.21.15.195:0/3079487046 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fcad002ff60 0x7fcad0032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:12.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:12.111+0000 7fcaebe3a700 1 --2- 172.21.15.195:0/3079487046 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcae40f4740 0x7fcae40f72f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:12.113 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:12.111+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3079487046 >> 172.21.15.195:0/3079487046 conn(0x7fcae40f0260 msgr2=0x7fcae40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:12.113 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:12.111+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3079487046 shutdown_connections 2023-10-12T22:38:12.113 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:12.111+0000 7fcaebe3a700 1 -- 172.21.15.195:0/3079487046 wait complete. 2023-10-12T22:38:12.113 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:12.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:10.860640+0000 mgr.smithi131.efktma (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:13.447 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:13.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:13.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:13 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:12.109319+0000 mon.smithi131 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.195:0/3079487046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:14.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:14 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:12.860987+0000 mgr.smithi131.efktma (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:14.550 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.548+0000 7fe7fc594700 1 -- 172.21.15.195:0/2652618723 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7f40f6510 msgr2=0x7fe7f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:14.551 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.548+0000 7fe7fc594700 1 --2- 172.21.15.195:0/2652618723 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7f40f6510 0x7fe7f40f6930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fe7f0009fb0 tx=0x7fe7f0009b30).stop 2023-10-12T22:38:14.551 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.548+0000 7fe7fc594700 1 -- 172.21.15.195:0/2652618723 shutdown_connections 2023-10-12T22:38:14.551 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.548+0000 7fe7fc594700 1 --2- 172.21.15.195:0/2652618723 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7f40f6510 0x7fe7f40f6930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:14.551 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.548+0000 7fe7fc594700 1 -- 172.21.15.195:0/2652618723 >> 172.21.15.195:0/2652618723 conn(0x7fe7f40f1c70 msgr2=0x7fe7f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:14.552 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.548+0000 7fe7fc594700 1 -- 172.21.15.195:0/2652618723 shutdown_connections 2023-10-12T22:38:14.552 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.548+0000 7fe7fc594700 1 -- 172.21.15.195:0/2652618723 wait complete. 2023-10-12T22:38:14.552 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.549+0000 7fe7fc594700 1 Processor -- start 2023-10-12T22:38:14.552 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.549+0000 7fe7fc594700 1 -- start start 2023-10-12T22:38:14.552 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.549+0000 7fe7fc594700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7f40f6510 0x7fe7f406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:14.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.549+0000 7fe7fc594700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f4069e20 con 0x7fe7f40f6510 2023-10-12T22:38:14.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.549+0000 7fe7fa330700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7f40f6510 0x7fe7f406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:14.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.549+0000 7fe7fa330700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7f40f6510 0x7fe7f406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:46054/0 (socket says 172.21.15.195:46054) 2023-10-12T22:38:14.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.549+0000 7fe7fa330700 1 -- 172.21.15.195:0/589523100 learned_addr learned my addr 172.21.15.195:0/589523100 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:14.553 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.549+0000 7fe7fa330700 1 -- 172.21.15.195:0/589523100 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f0005040 con 0x7fe7f40f6510 2023-10-12T22:38:14.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.550+0000 7fe7fa330700 1 --2- 172.21.15.195:0/589523100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7f40f6510 0x7fe7f406b9a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fe7f0013510 tx=0x7fe7f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:14.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.550+0000 7fe7e6ffd700 1 -- 172.21.15.195:0/589523100 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f0009320 con 0x7fe7f40f6510 2023-10-12T22:38:14.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.550+0000 7fe7fc594700 1 -- 172.21.15.195:0/589523100 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f4069fc0 con 0x7fe7f40f6510 2023-10-12T22:38:14.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.550+0000 7fe7e6ffd700 1 -- 172.21.15.195:0/589523100 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7f0013f00 con 0x7fe7f40f6510 2023-10-12T22:38:14.554 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.550+0000 7fe7e6ffd700 1 -- 172.21.15.195:0/589523100 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f0014d90 con 0x7fe7f40f6510 2023-10-12T22:38:14.555 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.550+0000 7fe7fc594700 1 -- 172.21.15.195:0/589523100 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f406a450 con 0x7fe7f40f6510 2023-10-12T22:38:14.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.551+0000 7fe7e6ffd700 1 -- 172.21.15.195:0/589523100 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7f00095f0 con 0x7fe7f40f6510 2023-10-12T22:38:14.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.551+0000 7fe7fc594700 1 -- 172.21.15.195:0/589523100 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f4059100 con 0x7fe7f40f6510 2023-10-12T22:38:14.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.551+0000 7fe7e6ffd700 1 --2- 172.21.15.195:0/589523100 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe7e002ffb0 0x7fe7e0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:14.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.551+0000 7fe7e6ffd700 1 -- 172.21.15.195:0/589523100 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7f0040fe0 con 0x7fe7f40f6510 2023-10-12T22:38:14.556 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.551+0000 7fe7f9b2f700 1 --2- 172.21.15.195:0/589523100 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe7e002ffb0 0x7fe7e0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:14.557 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.551+0000 7fe7f9b2f700 1 --2- 172.21.15.195:0/589523100 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe7e002ffb0 0x7fe7e0032470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fe7e80096a0 tx=0x7fe7e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:14.557 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.556+0000 7fe7e6ffd700 1 -- 172.21.15.195:0/589523100 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7f006d2b0 con 0x7fe7f40f6510 2023-10-12T22:38:14.765 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.764+0000 7fe7fc594700 1 -- 172.21.15.195:0/589523100 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7f40fbf60 con 0x7fe7f40f6510 2023-10-12T22:38:14.765 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.765+0000 7fe7e6ffd700 1 -- 172.21.15.195:0/589523100 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe7f0014420 con 0x7fe7f40f6510 2023-10-12T22:38:14.765 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:14.766 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:14.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.766+0000 7fe7fc594700 1 -- 172.21.15.195:0/589523100 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe7e002ffb0 msgr2=0x7fe7e0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:14.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.766+0000 7fe7fc594700 1 --2- 172.21.15.195:0/589523100 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe7e002ffb0 0x7fe7e0032470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fe7e80096a0 tx=0x7fe7e8006b40).stop 2023-10-12T22:38:14.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.766+0000 7fe7fc594700 1 -- 172.21.15.195:0/589523100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7f40f6510 msgr2=0x7fe7f406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:14.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.766+0000 7fe7fc594700 1 --2- 172.21.15.195:0/589523100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7f40f6510 0x7fe7f406b9a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fe7f0013510 tx=0x7fe7f000b040).stop 2023-10-12T22:38:14.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.766+0000 7fe7fc594700 1 -- 172.21.15.195:0/589523100 shutdown_connections 2023-10-12T22:38:14.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.766+0000 7fe7fc594700 1 --2- 172.21.15.195:0/589523100 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe7e002ffb0 0x7fe7e0032470 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:14.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.766+0000 7fe7fc594700 1 --2- 172.21.15.195:0/589523100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7f40f6510 0x7fe7f406b9a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:14.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.766+0000 7fe7fc594700 1 -- 172.21.15.195:0/589523100 >> 172.21.15.195:0/589523100 conn(0x7fe7f40f1c70 msgr2=0x7fe7f40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:14.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.766+0000 7fe7fc594700 1 -- 172.21.15.195:0/589523100 shutdown_connections 2023-10-12T22:38:14.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:14.766+0000 7fe7fc594700 1 -- 172.21.15.195:0/589523100 wait complete. 2023-10-12T22:38:14.769 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:15.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:15 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:14.765000+0000 mon.smithi131 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.195:0/589523100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:16.268 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:16.268 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:16.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:16 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:14.861320+0000 mgr.smithi131.efktma (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:17.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.426+0000 7f7ad6492700 1 -- 172.21.15.195:0/2394837168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad00f6510 msgr2=0x7f7ad00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:17.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.426+0000 7f7ad6492700 1 --2- 172.21.15.195:0/2394837168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad00f6510 0x7f7ad00f6930 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f7ac0009fb0 tx=0x7f7ac0009b30).stop 2023-10-12T22:38:17.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.426+0000 7f7ad6492700 1 -- 172.21.15.195:0/2394837168 shutdown_connections 2023-10-12T22:38:17.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.426+0000 7f7ad6492700 1 --2- 172.21.15.195:0/2394837168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad00f6510 0x7f7ad00f6930 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:17.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.426+0000 7f7ad6492700 1 -- 172.21.15.195:0/2394837168 >> 172.21.15.195:0/2394837168 conn(0x7f7ad00f1c70 msgr2=0x7f7ad00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:17.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.426+0000 7f7ad6492700 1 -- 172.21.15.195:0/2394837168 shutdown_connections 2023-10-12T22:38:17.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.426+0000 7f7ad6492700 1 -- 172.21.15.195:0/2394837168 wait complete. 2023-10-12T22:38:17.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.427+0000 7f7ad6492700 1 Processor -- start 2023-10-12T22:38:17.429 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.427+0000 7f7ad6492700 1 -- start start 2023-10-12T22:38:17.429 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.427+0000 7f7ad6492700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad006b960 0x7f7ad0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:17.429 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.427+0000 7f7ad6492700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad006bd80 con 0x7f7ad006b960 2023-10-12T22:38:17.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.427+0000 7f7acffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad006b960 0x7f7ad0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:17.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.427+0000 7f7acffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad006b960 0x7f7ad0069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:46074/0 (socket says 172.21.15.195:46074) 2023-10-12T22:38:17.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.427+0000 7f7acffff700 1 -- 172.21.15.195:0/3429963220 learned_addr learned my addr 172.21.15.195:0/3429963220 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:17.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.428+0000 7f7acffff700 1 -- 172.21.15.195:0/3429963220 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac0005040 con 0x7f7ad006b960 2023-10-12T22:38:17.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.428+0000 7f7acffff700 1 --2- 172.21.15.195:0/3429963220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad006b960 0x7f7ad0069e20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f7ac0009f80 tx=0x7f7ac0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:17.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.428+0000 7f7accff9700 1 -- 172.21.15.195:0/3429963220 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ac00136c0 con 0x7f7ad006b960 2023-10-12T22:38:17.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.428+0000 7f7ad6492700 1 -- 172.21.15.195:0/3429963220 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad006a310 con 0x7f7ad006b960 2023-10-12T22:38:17.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.428+0000 7f7accff9700 1 -- 172.21.15.195:0/3429963220 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ac0013e70 con 0x7f7ad006b960 2023-10-12T22:38:17.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.428+0000 7f7accff9700 1 -- 172.21.15.195:0/3429963220 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ac0014420 con 0x7f7ad006b960 2023-10-12T22:38:17.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.428+0000 7f7ad6492700 1 -- 172.21.15.195:0/3429963220 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad006a800 con 0x7f7ad006b960 2023-10-12T22:38:17.433 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.429+0000 7f7accff9700 1 -- 172.21.15.195:0/3429963220 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ac0025060 con 0x7f7ad006b960 2023-10-12T22:38:17.433 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.429+0000 7f7accff9700 1 --2- 172.21.15.195:0/3429963220 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7ab802ff60 0x7f7ab8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:17.433 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.429+0000 7f7ad6492700 1 -- 172.21.15.195:0/3429963220 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad0059100 con 0x7f7ad006b960 2023-10-12T22:38:17.433 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.429+0000 7f7accff9700 1 -- 172.21.15.195:0/3429963220 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ac0040ea0 con 0x7f7ad006b960 2023-10-12T22:38:17.433 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.429+0000 7f7acf7fe700 1 --2- 172.21.15.195:0/3429963220 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7ab802ff60 0x7f7ab8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:17.434 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.430+0000 7f7acf7fe700 1 --2- 172.21.15.195:0/3429963220 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7ab802ff60 0x7f7ab8032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7ac40096a0 tx=0x7f7ac4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:17.434 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.434+0000 7f7accff9700 1 -- 172.21.15.195:0/3429963220 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ac0014580 con 0x7f7ad006b960 2023-10-12T22:38:17.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.637+0000 7f7ad6492700 1 -- 172.21.15.195:0/3429963220 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ad006abf0 con 0x7f7ad006b960 2023-10-12T22:38:17.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.638+0000 7f7accff9700 1 -- 172.21.15.195:0/3429963220 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7ac0004490 con 0x7f7ad006b960 2023-10-12T22:38:17.639 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:17.640 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.640+0000 7f7ad6492700 1 -- 172.21.15.195:0/3429963220 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7ab802ff60 msgr2=0x7f7ab8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.640+0000 7f7ad6492700 1 --2- 172.21.15.195:0/3429963220 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7ab802ff60 0x7f7ab8032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7ac40096a0 tx=0x7f7ac4006b40).stop 2023-10-12T22:38:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.640+0000 7f7ad6492700 1 -- 172.21.15.195:0/3429963220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad006b960 msgr2=0x7f7ad0069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.640+0000 7f7ad6492700 1 --2- 172.21.15.195:0/3429963220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad006b960 0x7f7ad0069e20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f7ac0009f80 tx=0x7f7ac0009b30).stop 2023-10-12T22:38:17.642 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.640+0000 7f7ad6492700 1 -- 172.21.15.195:0/3429963220 shutdown_connections 2023-10-12T22:38:17.642 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.640+0000 7f7ad6492700 1 --2- 172.21.15.195:0/3429963220 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7ab802ff60 0x7f7ab8032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:17.642 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.640+0000 7f7ad6492700 1 --2- 172.21.15.195:0/3429963220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ad006b960 0x7f7ad0069e20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:17.642 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.640+0000 7f7ad6492700 1 -- 172.21.15.195:0/3429963220 >> 172.21.15.195:0/3429963220 conn(0x7f7ad00f1c70 msgr2=0x7f7ad00f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:17.642 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.640+0000 7f7ad6492700 1 -- 172.21.15.195:0/3429963220 shutdown_connections 2023-10-12T22:38:17.643 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:17.640+0000 7f7ad6492700 1 -- 172.21.15.195:0/3429963220 wait complete. 2023-10-12T22:38:17.643 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:18.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:18 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:16.861675+0000 mgr.smithi131.efktma (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:18.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:18 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:17.639318+0000 mon.smithi131 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.195:0/3429963220' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:19.024 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:19.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:20.207 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.206+0000 7f1eb9634700 1 -- 172.21.15.195:0/3740909260 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1eb40f6510 msgr2=0x7f1eb40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:20.208 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.206+0000 7f1eb9634700 1 --2- 172.21.15.195:0/3740909260 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1eb40f6510 0x7f1eb40f6930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f1ea4009fb0 tx=0x7f1ea4009b30).stop 2023-10-12T22:38:20.208 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.206+0000 7f1eb9634700 1 -- 172.21.15.195:0/3740909260 shutdown_connections 2023-10-12T22:38:20.208 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.206+0000 7f1eb9634700 1 --2- 172.21.15.195:0/3740909260 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1eb40f6510 0x7f1eb40f6930 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:20.208 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.206+0000 7f1eb9634700 1 -- 172.21.15.195:0/3740909260 >> 172.21.15.195:0/3740909260 conn(0x7f1eb40f1c70 msgr2=0x7f1eb40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:20.208 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.206+0000 7f1eb9634700 1 -- 172.21.15.195:0/3740909260 shutdown_connections 2023-10-12T22:38:20.209 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.206+0000 7f1eb9634700 1 -- 172.21.15.195:0/3740909260 wait complete. 2023-10-12T22:38:20.211 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.207+0000 7f1eb9634700 1 Processor -- start 2023-10-12T22:38:20.211 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.207+0000 7f1eb9634700 1 -- start start 2023-10-12T22:38:20.211 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.207+0000 7f1eb9634700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1eb40f6510 0x7f1eb4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:20.211 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.207+0000 7f1eb9634700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb4106c20 con 0x7f1eb40f6510 2023-10-12T22:38:20.211 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.207+0000 7f1eb2ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1eb40f6510 0x7f1eb4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:20.212 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.207+0000 7f1eb2ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1eb40f6510 0x7f1eb4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:46458/0 (socket says 172.21.15.195:46458) 2023-10-12T22:38:20.212 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.207+0000 7f1eb2ffd700 1 -- 172.21.15.195:0/3468182164 learned_addr learned my addr 172.21.15.195:0/3468182164 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:20.212 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.208+0000 7f1eb2ffd700 1 -- 172.21.15.195:0/3468182164 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea4005040 con 0x7f1eb40f6510 2023-10-12T22:38:20.212 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.208+0000 7f1eb2ffd700 1 --2- 172.21.15.195:0/3468182164 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1eb40f6510 0x7f1eb4106730 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f1ea4013510 tx=0x7f1ea400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:20.212 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.208+0000 7f1eab7fe700 1 -- 172.21.15.195:0/3468182164 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ea4009320 con 0x7f1eb40f6510 2023-10-12T22:38:20.213 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.208+0000 7f1eb9634700 1 -- 172.21.15.195:0/3468182164 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb4106e20 con 0x7f1eb40f6510 2023-10-12T22:38:20.213 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.208+0000 7f1eab7fe700 1 -- 172.21.15.195:0/3468182164 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ea4013f00 con 0x7f1eb40f6510 2023-10-12T22:38:20.213 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.208+0000 7f1eab7fe700 1 -- 172.21.15.195:0/3468182164 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ea4014d90 con 0x7f1eb40f6510 2023-10-12T22:38:20.213 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.208+0000 7f1eb9634700 1 -- 172.21.15.195:0/3468182164 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb41072b0 con 0x7f1eb40f6510 2023-10-12T22:38:20.213 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.209+0000 7f1eab7fe700 1 -- 172.21.15.195:0/3468182164 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ea4014420 con 0x7f1eb40f6510 2023-10-12T22:38:20.213 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.209+0000 7f1eb9634700 1 -- 172.21.15.195:0/3468182164 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb4045ad0 con 0x7f1eb40f6510 2023-10-12T22:38:20.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.209+0000 7f1eab7fe700 1 --2- 172.21.15.195:0/3468182164 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1ea002ff60 0x7f1ea0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:20.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.209+0000 7f1eab7fe700 1 -- 172.21.15.195:0/3468182164 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ea4041070 con 0x7f1eb40f6510 2023-10-12T22:38:20.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.209+0000 7f1eb27fc700 1 --2- 172.21.15.195:0/3468182164 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1ea002ff60 0x7f1ea0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:20.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.210+0000 7f1eb27fc700 1 --2- 172.21.15.195:0/3468182164 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1ea002ff60 0x7f1ea0032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f1e9c0096a0 tx=0x7f1e9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:20.215 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.214+0000 7f1eab7fe700 1 -- 172.21.15.195:0/3468182164 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ea4026bf0 con 0x7f1eb40f6510 2023-10-12T22:38:20.423 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.422+0000 7f1eb9634700 1 -- 172.21.15.195:0/3468182164 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1eb40fbf60 con 0x7f1eb40f6510 2023-10-12T22:38:20.424 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.422+0000 7f1eab7fe700 1 -- 172.21.15.195:0/3468182164 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1ea401db70 con 0x7f1eb40f6510 2023-10-12T22:38:20.424 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:20.425 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:20.425 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.424+0000 7f1eb9634700 1 -- 172.21.15.195:0/3468182164 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1ea002ff60 msgr2=0x7f1ea0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:20.426 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.424+0000 7f1eb9634700 1 --2- 172.21.15.195:0/3468182164 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1ea002ff60 0x7f1ea0032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f1e9c0096a0 tx=0x7f1e9c006b40).stop 2023-10-12T22:38:20.426 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.424+0000 7f1eb9634700 1 -- 172.21.15.195:0/3468182164 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1eb40f6510 msgr2=0x7f1eb4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:20.426 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.424+0000 7f1eb9634700 1 --2- 172.21.15.195:0/3468182164 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1eb40f6510 0x7f1eb4106730 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f1ea4013510 tx=0x7f1ea400b040).stop 2023-10-12T22:38:20.426 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.424+0000 7f1eb9634700 1 -- 172.21.15.195:0/3468182164 shutdown_connections 2023-10-12T22:38:20.426 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.424+0000 7f1eb9634700 1 --2- 172.21.15.195:0/3468182164 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1ea002ff60 0x7f1ea0032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:20.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.424+0000 7f1eb9634700 1 --2- 172.21.15.195:0/3468182164 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1eb40f6510 0x7f1eb4106730 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:20.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.424+0000 7f1eb9634700 1 -- 172.21.15.195:0/3468182164 >> 172.21.15.195:0/3468182164 conn(0x7f1eb40f1c70 msgr2=0x7f1eb40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:20.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.424+0000 7f1eb9634700 1 -- 172.21.15.195:0/3468182164 shutdown_connections 2023-10-12T22:38:20.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:20.424+0000 7f1eb9634700 1 -- 172.21.15.195:0/3468182164 wait complete. 2023-10-12T22:38:20.428 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:20.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:20 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:18.862046+0000 mgr.smithi131.efktma (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:21.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:21 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:20.423857+0000 mon.smithi131 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.195:0/3468182164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:21.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:21.814 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:22.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:20.862363+0000 mgr.smithi131.efktma (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:22.965 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.963+0000 7fdf72f89700 1 -- 172.21.15.195:0/3318219532 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf6c0f5410 msgr2=0x7fdf6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:22.966 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.963+0000 7fdf72f89700 1 --2- 172.21.15.195:0/3318219532 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c0f5830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fdf60009fb0 tx=0x7fdf60009b30).stop 2023-10-12T22:38:22.966 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.963+0000 7fdf72f89700 1 -- 172.21.15.195:0/3318219532 shutdown_connections 2023-10-12T22:38:22.966 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.963+0000 7fdf72f89700 1 --2- 172.21.15.195:0/3318219532 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c0f5830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:22.966 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.963+0000 7fdf72f89700 1 -- 172.21.15.195:0/3318219532 >> 172.21.15.195:0/3318219532 conn(0x7fdf6c0f0b70 msgr2=0x7fdf6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:22.967 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.963+0000 7fdf72f89700 1 -- 172.21.15.195:0/3318219532 shutdown_connections 2023-10-12T22:38:22.967 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.963+0000 7fdf72f89700 1 -- 172.21.15.195:0/3318219532 wait complete. 2023-10-12T22:38:22.967 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.963+0000 7fdf72f89700 1 Processor -- start 2023-10-12T22:38:22.967 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.963+0000 7fdf72f89700 1 -- start start 2023-10-12T22:38:22.967 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.964+0000 7fdf72f89700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:22.968 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.964+0000 7fdf72f89700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c060220 con 0x7fdf6c0f5410 2023-10-12T22:38:22.968 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.964+0000 7fdf70d25700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:22.968 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.964+0000 7fdf70d25700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:46486/0 (socket says 172.21.15.195:46486) 2023-10-12T22:38:22.968 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.964+0000 7fdf70d25700 1 -- 172.21.15.195:0/1124097099 learned_addr learned my addr 172.21.15.195:0/1124097099 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:22.968 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.964+0000 7fdf70d25700 1 -- 172.21.15.195:0/1124097099 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf60005040 con 0x7fdf6c0f5410 2023-10-12T22:38:22.969 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.964+0000 7fdf70d25700 1 --2- 172.21.15.195:0/1124097099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c109450 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fdf60014000 tx=0x7fdf6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:22.969 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.964+0000 7fdf697fa700 1 -- 172.21.15.195:0/1124097099 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf600095f0 con 0x7fdf6c0f5410 2023-10-12T22:38:22.969 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.964+0000 7fdf72f89700 1 -- 172.21.15.195:0/1124097099 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6c060420 con 0x7fdf6c0f5410 2023-10-12T22:38:22.969 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.964+0000 7fdf697fa700 1 -- 172.21.15.195:0/1124097099 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf60007e70 con 0x7fdf6c0f5410 2023-10-12T22:38:22.969 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.964+0000 7fdf697fa700 1 -- 172.21.15.195:0/1124097099 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf60014ea0 con 0x7fdf6c0f5410 2023-10-12T22:38:22.970 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.964+0000 7fdf72f89700 1 -- 172.21.15.195:0/1124097099 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6c109b00 con 0x7fdf6c0f5410 2023-10-12T22:38:22.971 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.965+0000 7fdf697fa700 1 -- 172.21.15.195:0/1124097099 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf60025070 con 0x7fdf6c0f5410 2023-10-12T22:38:22.971 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.965+0000 7fdf72f89700 1 -- 172.21.15.195:0/1124097099 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6c059100 con 0x7fdf6c0f5410 2023-10-12T22:38:22.971 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.965+0000 7fdf697fa700 1 --2- 172.21.15.195:0/1124097099 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf5c02ff60 0x7fdf5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:22.971 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.965+0000 7fdf697fa700 1 -- 172.21.15.195:0/1124097099 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf60040a10 con 0x7fdf6c0f5410 2023-10-12T22:38:22.972 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.966+0000 7fdf6bfff700 1 --2- 172.21.15.195:0/1124097099 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf5c02ff60 0x7fdf5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:22.972 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.966+0000 7fdf6bfff700 1 --2- 172.21.15.195:0/1124097099 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf5c02ff60 0x7fdf5c032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fdf580096a0 tx=0x7fdf58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:22.972 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:22.970+0000 7fdf697fa700 1 -- 172.21.15.195:0/1124097099 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf6000f4e0 con 0x7fdf6c0f5410 2023-10-12T22:38:23.176 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:23.175+0000 7fdf72f89700 1 -- 172.21.15.195:0/1124097099 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf6c045ad0 con 0x7fdf6c0f5410 2023-10-12T22:38:23.177 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:23.176+0000 7fdf697fa700 1 -- 172.21.15.195:0/1124097099 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdf6000f300 con 0x7fdf6c0f5410 2023-10-12T22:38:23.177 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:23.178 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:23.179 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:23.177+0000 7fdf72f89700 1 -- 172.21.15.195:0/1124097099 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf5c02ff60 msgr2=0x7fdf5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:23.179 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:23.177+0000 7fdf72f89700 1 --2- 172.21.15.195:0/1124097099 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf5c02ff60 0x7fdf5c032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fdf580096a0 tx=0x7fdf58006b40).stop 2023-10-12T22:38:23.179 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:23.177+0000 7fdf72f89700 1 -- 172.21.15.195:0/1124097099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf6c0f5410 msgr2=0x7fdf6c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:23.179 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:23.177+0000 7fdf72f89700 1 --2- 172.21.15.195:0/1124097099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c109450 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fdf60014000 tx=0x7fdf6000b040).stop 2023-10-12T22:38:23.180 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:23.177+0000 7fdf72f89700 1 -- 172.21.15.195:0/1124097099 shutdown_connections 2023-10-12T22:38:23.180 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:23.177+0000 7fdf72f89700 1 --2- 172.21.15.195:0/1124097099 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdf5c02ff60 0x7fdf5c032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:23.180 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:23.177+0000 7fdf72f89700 1 --2- 172.21.15.195:0/1124097099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf6c0f5410 0x7fdf6c109450 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:23.180 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:23.177+0000 7fdf72f89700 1 -- 172.21.15.195:0/1124097099 >> 172.21.15.195:0/1124097099 conn(0x7fdf6c0f0b70 msgr2=0x7fdf6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:23.180 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:23.177+0000 7fdf72f89700 1 -- 172.21.15.195:0/1124097099 shutdown_connections 2023-10-12T22:38:23.181 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:23.177+0000 7fdf72f89700 1 -- 172.21.15.195:0/1124097099 wait complete. 2023-10-12T22:38:23.181 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:23.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:22.938592+0000 mon.smithi131 (mon.0) 278 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/mirror_snapshot_schedule"}]: dispatch 2023-10-12T22:38:23.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:22.943488+0000 mon.smithi131 (mon.0) 279 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/trash_purge_schedule"}]: dispatch 2023-10-12T22:38:24.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:22.862583+0000 mgr.smithi131.efktma (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:24.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:23.177048+0000 mon.smithi131 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.195:0/1124097099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:24.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:24.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:25.691 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.689+0000 7f63c0a45700 1 -- 172.21.15.195:0/2443064839 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63bc0f62a0 msgr2=0x7f63bc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:25.692 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.689+0000 7f63c0a45700 1 --2- 172.21.15.195:0/2443064839 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63bc0f62a0 0x7f63bc0f66c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f63ac009fb0 tx=0x7f63ac009b30).stop 2023-10-12T22:38:25.692 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.690+0000 7f63c0a45700 1 -- 172.21.15.195:0/2443064839 shutdown_connections 2023-10-12T22:38:25.692 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.690+0000 7f63c0a45700 1 --2- 172.21.15.195:0/2443064839 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63bc0f62a0 0x7f63bc0f66c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:25.692 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.690+0000 7f63c0a45700 1 -- 172.21.15.195:0/2443064839 >> 172.21.15.195:0/2443064839 conn(0x7f63bc0f1aa0 msgr2=0x7f63bc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:25.692 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.690+0000 7f63c0a45700 1 -- 172.21.15.195:0/2443064839 shutdown_connections 2023-10-12T22:38:25.693 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.690+0000 7f63c0a45700 1 -- 172.21.15.195:0/2443064839 wait complete. 2023-10-12T22:38:25.693 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.690+0000 7f63c0a45700 1 Processor -- start 2023-10-12T22:38:25.693 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.690+0000 7f63c0a45700 1 -- start start 2023-10-12T22:38:25.693 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.690+0000 7f63c0a45700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63bc0f62a0 0x7f63bc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:25.693 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.690+0000 7f63c0a45700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc069e20 con 0x7f63bc0f62a0 2023-10-12T22:38:25.695 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.691+0000 7f63bb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63bc0f62a0 0x7f63bc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:25.695 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.691+0000 7f63bb7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63bc0f62a0 0x7f63bc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:46512/0 (socket says 172.21.15.195:46512) 2023-10-12T22:38:25.695 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.691+0000 7f63bb7fe700 1 -- 172.21.15.195:0/58629535 learned_addr learned my addr 172.21.15.195:0/58629535 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:25.696 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.691+0000 7f63bb7fe700 1 -- 172.21.15.195:0/58629535 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac005040 con 0x7f63bc0f62a0 2023-10-12T22:38:25.696 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.691+0000 7f63bb7fe700 1 --2- 172.21.15.195:0/58629535 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63bc0f62a0 0x7f63bc06b9a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f63ac013510 tx=0x7f63ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:25.696 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.691+0000 7f63a3fff700 1 -- 172.21.15.195:0/58629535 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63ac009320 con 0x7f63bc0f62a0 2023-10-12T22:38:25.696 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.691+0000 7f63c0a45700 1 -- 172.21.15.195:0/58629535 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63bc069fc0 con 0x7f63bc0f62a0 2023-10-12T22:38:25.696 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.691+0000 7f63a3fff700 1 -- 172.21.15.195:0/58629535 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63ac013f00 con 0x7f63bc0f62a0 2023-10-12T22:38:25.697 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.691+0000 7f63a3fff700 1 -- 172.21.15.195:0/58629535 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63ac014dc0 con 0x7f63bc0f62a0 2023-10-12T22:38:25.697 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.691+0000 7f63c0a45700 1 -- 172.21.15.195:0/58629535 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63bc06a450 con 0x7f63bc0f62a0 2023-10-12T22:38:25.697 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.692+0000 7f63a3fff700 1 -- 172.21.15.195:0/58629535 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63ac014420 con 0x7f63bc0f62a0 2023-10-12T22:38:25.697 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.692+0000 7f63c0a45700 1 -- 172.21.15.195:0/58629535 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63bc059100 con 0x7f63bc0f62a0 2023-10-12T22:38:25.697 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.692+0000 7f63a3fff700 1 --2- 172.21.15.195:0/58629535 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f63a402ff60 0x7f63a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:25.698 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.692+0000 7f63a3fff700 1 -- 172.21.15.195:0/58629535 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63ac041020 con 0x7f63bc0f62a0 2023-10-12T22:38:25.698 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.692+0000 7f63baffd700 1 --2- 172.21.15.195:0/58629535 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f63a402ff60 0x7f63a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:25.698 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.693+0000 7f63baffd700 1 --2- 172.21.15.195:0/58629535 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f63a402ff60 0x7f63a4032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f63b00096a0 tx=0x7f63b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:25.698 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.697+0000 7f63a3fff700 1 -- 172.21.15.195:0/58629535 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63ac0095f0 con 0x7f63bc0f62a0 2023-10-12T22:38:25.904 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.902+0000 7f63c0a45700 1 -- 172.21.15.195:0/58629535 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63bc045ad0 con 0x7f63bc0f62a0 2023-10-12T22:38:25.904 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.903+0000 7f63a3fff700 1 -- 172.21.15.195:0/58629535 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f63ac01cb10 con 0x7f63bc0f62a0 2023-10-12T22:38:25.905 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:25.905 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:25.906 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.904+0000 7f63c0a45700 1 -- 172.21.15.195:0/58629535 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f63a402ff60 msgr2=0x7f63a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:25.906 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.904+0000 7f63c0a45700 1 --2- 172.21.15.195:0/58629535 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f63a402ff60 0x7f63a4032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f63b00096a0 tx=0x7f63b0006b40).stop 2023-10-12T22:38:25.906 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.904+0000 7f63c0a45700 1 -- 172.21.15.195:0/58629535 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63bc0f62a0 msgr2=0x7f63bc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:25.906 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.904+0000 7f63c0a45700 1 --2- 172.21.15.195:0/58629535 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63bc0f62a0 0x7f63bc06b9a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f63ac013510 tx=0x7f63ac00b040).stop 2023-10-12T22:38:25.907 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.904+0000 7f63c0a45700 1 -- 172.21.15.195:0/58629535 shutdown_connections 2023-10-12T22:38:25.907 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.904+0000 7f63c0a45700 1 --2- 172.21.15.195:0/58629535 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f63a402ff60 0x7f63a4032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:25.907 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.904+0000 7f63c0a45700 1 --2- 172.21.15.195:0/58629535 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63bc0f62a0 0x7f63bc06b9a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:25.907 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.904+0000 7f63c0a45700 1 -- 172.21.15.195:0/58629535 >> 172.21.15.195:0/58629535 conn(0x7f63bc0f1aa0 msgr2=0x7f63bc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:25.907 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.905+0000 7f63c0a45700 1 -- 172.21.15.195:0/58629535 shutdown_connections 2023-10-12T22:38:25.907 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:25.905+0000 7f63c0a45700 1 -- 172.21.15.195:0/58629535 wait complete. 2023-10-12T22:38:25.908 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:26.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:26 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:24.862823+0000 mgr.smithi131.efktma (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:26.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:26 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:25.904355+0000 mon.smithi131 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.195:0/58629535' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:27.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:27.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:28.448 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.446+0000 7fc3b95cf700 1 -- 172.21.15.195:0/552093630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc3b40f5410 msgr2=0x7fc3b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:28.448 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.446+0000 7fc3b95cf700 1 --2- 172.21.15.195:0/552093630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc3b40f5410 0x7fc3b40f5830 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fc3a8009fb0 tx=0x7fc3a8009b30).stop 2023-10-12T22:38:28.449 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.446+0000 7fc3b95cf700 1 -- 172.21.15.195:0/552093630 shutdown_connections 2023-10-12T22:38:28.449 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.446+0000 7fc3b95cf700 1 --2- 172.21.15.195:0/552093630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc3b40f5410 0x7fc3b40f5830 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:28.449 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.446+0000 7fc3b95cf700 1 -- 172.21.15.195:0/552093630 >> 172.21.15.195:0/552093630 conn(0x7fc3b40f0b70 msgr2=0x7fc3b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:28.449 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.446+0000 7fc3b95cf700 1 -- 172.21.15.195:0/552093630 shutdown_connections 2023-10-12T22:38:28.449 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.447+0000 7fc3b95cf700 1 -- 172.21.15.195:0/552093630 wait complete. 2023-10-12T22:38:28.452 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.447+0000 7fc3b95cf700 1 Processor -- start 2023-10-12T22:38:28.452 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.447+0000 7fc3b95cf700 1 -- start start 2023-10-12T22:38:28.452 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.447+0000 7fc3b95cf700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc3b40f5410 0x7fc3b4106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:28.452 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.447+0000 7fc3b95cf700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b4106bf0 con 0x7fc3b40f5410 2023-10-12T22:38:28.452 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.448+0000 7fc3b2ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc3b40f5410 0x7fc3b4106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:28.453 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.448+0000 7fc3b2ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc3b40f5410 0x7fc3b4106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:55676/0 (socket says 172.21.15.195:55676) 2023-10-12T22:38:28.453 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.448+0000 7fc3b2ffd700 1 -- 172.21.15.195:0/3959735719 learned_addr learned my addr 172.21.15.195:0/3959735719 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:28.453 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.448+0000 7fc3b2ffd700 1 -- 172.21.15.195:0/3959735719 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a8005040 con 0x7fc3b40f5410 2023-10-12T22:38:28.453 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.448+0000 7fc3b2ffd700 1 --2- 172.21.15.195:0/3959735719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc3b40f5410 0x7fc3b4106700 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fc3a8013510 tx=0x7fc3a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:28.453 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.448+0000 7fc39b7fe700 1 -- 172.21.15.195:0/3959735719 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3a8009320 con 0x7fc3b40f5410 2023-10-12T22:38:28.454 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.448+0000 7fc3b95cf700 1 -- 172.21.15.195:0/3959735719 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b4106df0 con 0x7fc3b40f5410 2023-10-12T22:38:28.454 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.448+0000 7fc39b7fe700 1 -- 172.21.15.195:0/3959735719 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3a8013f00 con 0x7fc3b40f5410 2023-10-12T22:38:28.454 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.448+0000 7fc39b7fe700 1 -- 172.21.15.195:0/3959735719 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3a8014d90 con 0x7fc3b40f5410 2023-10-12T22:38:28.454 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.448+0000 7fc3b95cf700 1 -- 172.21.15.195:0/3959735719 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b4107280 con 0x7fc3b40f5410 2023-10-12T22:38:28.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.449+0000 7fc39b7fe700 1 -- 172.21.15.195:0/3959735719 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3a8014420 con 0x7fc3b40f5410 2023-10-12T22:38:28.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.449+0000 7fc3b95cf700 1 -- 172.21.15.195:0/3959735719 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b4045ad0 con 0x7fc3b40f5410 2023-10-12T22:38:28.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.449+0000 7fc39b7fe700 1 --2- 172.21.15.195:0/3959735719 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc39c02ff60 0x7fc39c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:28.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.449+0000 7fc39b7fe700 1 -- 172.21.15.195:0/3959735719 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc3a8041000 con 0x7fc3b40f5410 2023-10-12T22:38:28.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.449+0000 7fc3b27fc700 1 --2- 172.21.15.195:0/3959735719 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc39c02ff60 0x7fc39c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:28.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.450+0000 7fc3b27fc700 1 --2- 172.21.15.195:0/3959735719 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc39c02ff60 0x7fc39c032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc3a40096a0 tx=0x7fc3a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:28.456 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.454+0000 7fc39b7fe700 1 -- 172.21.15.195:0/3959735719 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3a80095f0 con 0x7fc3b40f5410 2023-10-12T22:38:28.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:28 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:26.863089+0000 mgr.smithi131.efktma (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:28.659 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.658+0000 7fc3b95cf700 1 -- 172.21.15.195:0/3959735719 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3b40fae60 con 0x7fc3b40f5410 2023-10-12T22:38:28.660 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.659+0000 7fc39b7fe700 1 -- 172.21.15.195:0/3959735719 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc3a801db70 con 0x7fc3b40f5410 2023-10-12T22:38:28.660 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:28.661 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:28.662 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.660+0000 7fc3b95cf700 1 -- 172.21.15.195:0/3959735719 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc39c02ff60 msgr2=0x7fc39c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:28.662 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.660+0000 7fc3b95cf700 1 --2- 172.21.15.195:0/3959735719 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc39c02ff60 0x7fc39c032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc3a40096a0 tx=0x7fc3a4006b40).stop 2023-10-12T22:38:28.662 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.661+0000 7fc3b95cf700 1 -- 172.21.15.195:0/3959735719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc3b40f5410 msgr2=0x7fc3b4106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:28.662 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.661+0000 7fc3b95cf700 1 --2- 172.21.15.195:0/3959735719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc3b40f5410 0x7fc3b4106700 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fc3a8013510 tx=0x7fc3a800b040).stop 2023-10-12T22:38:28.663 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.661+0000 7fc3b95cf700 1 -- 172.21.15.195:0/3959735719 shutdown_connections 2023-10-12T22:38:28.663 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.661+0000 7fc3b95cf700 1 --2- 172.21.15.195:0/3959735719 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc39c02ff60 0x7fc39c032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:28.663 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.661+0000 7fc3b95cf700 1 --2- 172.21.15.195:0/3959735719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc3b40f5410 0x7fc3b4106700 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:28.663 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.661+0000 7fc3b95cf700 1 -- 172.21.15.195:0/3959735719 >> 172.21.15.195:0/3959735719 conn(0x7fc3b40f0b70 msgr2=0x7fc3b40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:28.663 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.661+0000 7fc3b95cf700 1 -- 172.21.15.195:0/3959735719 shutdown_connections 2023-10-12T22:38:28.664 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:28.661+0000 7fc3b95cf700 1 -- 172.21.15.195:0/3959735719 wait complete. 2023-10-12T22:38:28.664 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:29.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:28.660483+0000 mon.smithi131 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.195:0/3959735719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:30.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:30.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:30.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:30 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:28.863417+0000 mgr.smithi131.efktma (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:31.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.266+0000 7fd14e818700 1 -- 172.21.15.195:0/3074419341 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1480f6510 msgr2=0x7fd1480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:31.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.266+0000 7fd14e818700 1 --2- 172.21.15.195:0/3074419341 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1480f6510 0x7fd1480f6930 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fd138009fb0 tx=0x7fd138009b30).stop 2023-10-12T22:38:31.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.266+0000 7fd14e818700 1 -- 172.21.15.195:0/3074419341 shutdown_connections 2023-10-12T22:38:31.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.266+0000 7fd14e818700 1 --2- 172.21.15.195:0/3074419341 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1480f6510 0x7fd1480f6930 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:31.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.266+0000 7fd14e818700 1 -- 172.21.15.195:0/3074419341 >> 172.21.15.195:0/3074419341 conn(0x7fd1480f1c70 msgr2=0x7fd1480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:31.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.266+0000 7fd14e818700 1 -- 172.21.15.195:0/3074419341 shutdown_connections 2023-10-12T22:38:31.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.266+0000 7fd14e818700 1 -- 172.21.15.195:0/3074419341 wait complete. 2023-10-12T22:38:31.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.267+0000 7fd14e818700 1 Processor -- start 2023-10-12T22:38:31.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.267+0000 7fd14e818700 1 -- start start 2023-10-12T22:38:31.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.267+0000 7fd14e818700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1480f6510 0x7fd148106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:31.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.267+0000 7fd14e818700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd148106c20 con 0x7fd1480f6510 2023-10-12T22:38:31.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.267+0000 7fd147fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1480f6510 0x7fd148106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:31.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.267+0000 7fd147fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1480f6510 0x7fd148106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:55696/0 (socket says 172.21.15.195:55696) 2023-10-12T22:38:31.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.267+0000 7fd147fff700 1 -- 172.21.15.195:0/3544262936 learned_addr learned my addr 172.21.15.195:0/3544262936 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:31.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.268+0000 7fd147fff700 1 -- 172.21.15.195:0/3544262936 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd138005040 con 0x7fd1480f6510 2023-10-12T22:38:31.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.268+0000 7fd147fff700 1 --2- 172.21.15.195:0/3544262936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1480f6510 0x7fd148106730 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fd138013510 tx=0x7fd13800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:31.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.268+0000 7fd144ff9700 1 -- 172.21.15.195:0/3544262936 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd138009320 con 0x7fd1480f6510 2023-10-12T22:38:31.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.268+0000 7fd14e818700 1 -- 172.21.15.195:0/3544262936 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd148106e20 con 0x7fd1480f6510 2023-10-12T22:38:31.273 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.268+0000 7fd144ff9700 1 -- 172.21.15.195:0/3544262936 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd138013f00 con 0x7fd1480f6510 2023-10-12T22:38:31.273 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.268+0000 7fd144ff9700 1 -- 172.21.15.195:0/3544262936 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd138014d90 con 0x7fd1480f6510 2023-10-12T22:38:31.273 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.268+0000 7fd14e818700 1 -- 172.21.15.195:0/3544262936 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1481072b0 con 0x7fd1480f6510 2023-10-12T22:38:31.274 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.269+0000 7fd144ff9700 1 -- 172.21.15.195:0/3544262936 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1380095f0 con 0x7fd1480f6510 2023-10-12T22:38:31.274 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.269+0000 7fd14e818700 1 -- 172.21.15.195:0/3544262936 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd148045ad0 con 0x7fd1480f6510 2023-10-12T22:38:31.275 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.269+0000 7fd144ff9700 1 --2- 172.21.15.195:0/3544262936 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd13002ff60 0x7fd130032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:31.275 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.269+0000 7fd144ff9700 1 -- 172.21.15.195:0/3544262936 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd1380410c0 con 0x7fd1480f6510 2023-10-12T22:38:31.275 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.269+0000 7fd1477fe700 1 --2- 172.21.15.195:0/3544262936 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd13002ff60 0x7fd130032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:31.275 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.270+0000 7fd1477fe700 1 --2- 172.21.15.195:0/3544262936 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd13002ff60 0x7fd130032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fd13c0096a0 tx=0x7fd13c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:31.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.274+0000 7fd144ff9700 1 -- 172.21.15.195:0/3544262936 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd138014420 con 0x7fd1480f6510 2023-10-12T22:38:31.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.481+0000 7fd14e818700 1 -- 172.21.15.195:0/3544262936 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd1480fbf60 con 0x7fd1480f6510 2023-10-12T22:38:31.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.482+0000 7fd144ff9700 1 -- 172.21.15.195:0/3544262936 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd13801d7f0 con 0x7fd1480f6510 2023-10-12T22:38:31.483 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:31.484 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:31.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.483+0000 7fd14e818700 1 -- 172.21.15.195:0/3544262936 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd13002ff60 msgr2=0x7fd130032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:31.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.483+0000 7fd14e818700 1 --2- 172.21.15.195:0/3544262936 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd13002ff60 0x7fd130032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fd13c0096a0 tx=0x7fd13c006b40).stop 2023-10-12T22:38:31.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.483+0000 7fd14e818700 1 -- 172.21.15.195:0/3544262936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1480f6510 msgr2=0x7fd148106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:31.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.483+0000 7fd14e818700 1 --2- 172.21.15.195:0/3544262936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1480f6510 0x7fd148106730 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fd138013510 tx=0x7fd13800b040).stop 2023-10-12T22:38:31.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.483+0000 7fd14e818700 1 -- 172.21.15.195:0/3544262936 shutdown_connections 2023-10-12T22:38:31.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.483+0000 7fd14e818700 1 --2- 172.21.15.195:0/3544262936 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd13002ff60 0x7fd130032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:31.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.483+0000 7fd14e818700 1 --2- 172.21.15.195:0/3544262936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1480f6510 0x7fd148106730 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:31.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.483+0000 7fd14e818700 1 -- 172.21.15.195:0/3544262936 >> 172.21.15.195:0/3544262936 conn(0x7fd1480f1c70 msgr2=0x7fd1480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:31.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.484+0000 7fd14e818700 1 -- 172.21.15.195:0/3544262936 shutdown_connections 2023-10-12T22:38:31.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:31.484+0000 7fd14e818700 1 -- 172.21.15.195:0/3544262936 wait complete. 2023-10-12T22:38:31.487 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:32.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:30.863725+0000 mgr.smithi131.efktma (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:32.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:31.483446+0000 mon.smithi131 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.195:0/3544262936' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:32.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:32.867 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:34.038 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.036+0000 7f4be631b700 1 -- 172.21.15.195:0/2574210033 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4be006b920 msgr2=0x7f4be0069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:34.038 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.036+0000 7f4be631b700 1 --2- 172.21.15.195:0/2574210033 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4be006b920 0x7f4be0069ec0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f4bdc009fb0 tx=0x7f4bdc009b30).stop 2023-10-12T22:38:34.038 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.036+0000 7f4be631b700 1 -- 172.21.15.195:0/2574210033 shutdown_connections 2023-10-12T22:38:34.039 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.036+0000 7f4be631b700 1 --2- 172.21.15.195:0/2574210033 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4be006b920 0x7f4be0069ec0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:34.039 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.036+0000 7f4be631b700 1 -- 172.21.15.195:0/2574210033 >> 172.21.15.195:0/2574210033 conn(0x7f4be00f1ac0 msgr2=0x7f4be00f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:34.039 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.036+0000 7f4be631b700 1 -- 172.21.15.195:0/2574210033 shutdown_connections 2023-10-12T22:38:34.039 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.036+0000 7f4be631b700 1 -- 172.21.15.195:0/2574210033 wait complete. 2023-10-12T22:38:34.039 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.037+0000 7f4be631b700 1 Processor -- start 2023-10-12T22:38:34.039 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.037+0000 7f4be631b700 1 -- start start 2023-10-12T22:38:34.040 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.037+0000 7f4be631b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4be006b920 0x7f4be010cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:34.040 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.037+0000 7f4be631b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be010cfb0 con 0x7f4be006b920 2023-10-12T22:38:34.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.037+0000 7f4be5319700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4be006b920 0x7f4be010cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:34.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.037+0000 7f4be5319700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4be006b920 0x7f4be010cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:55712/0 (socket says 172.21.15.195:55712) 2023-10-12T22:38:34.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.037+0000 7f4be5319700 1 -- 172.21.15.195:0/629141772 learned_addr learned my addr 172.21.15.195:0/629141772 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:34.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.037+0000 7f4be5319700 1 -- 172.21.15.195:0/629141772 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc005040 con 0x7f4be006b920 2023-10-12T22:38:34.042 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.037+0000 7f4be5319700 1 --2- 172.21.15.195:0/629141772 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4be006b920 0x7f4be010cac0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f4bdc014000 tx=0x7f4bdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:34.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.038+0000 7f4bd5ffb700 1 -- 172.21.15.195:0/629141772 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bdc0095f0 con 0x7f4be006b920 2023-10-12T22:38:34.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.038+0000 7f4be631b700 1 -- 172.21.15.195:0/629141772 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be010d1b0 con 0x7f4be006b920 2023-10-12T22:38:34.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.038+0000 7f4bd5ffb700 1 -- 172.21.15.195:0/629141772 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bdc007e70 con 0x7f4be006b920 2023-10-12T22:38:34.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.038+0000 7f4bd5ffb700 1 -- 172.21.15.195:0/629141772 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bdc014ea0 con 0x7f4be006b920 2023-10-12T22:38:34.043 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.038+0000 7f4be631b700 1 -- 172.21.15.195:0/629141772 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be010d640 con 0x7f4be006b920 2023-10-12T22:38:34.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.039+0000 7f4bd5ffb700 1 -- 172.21.15.195:0/629141772 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4bdc025070 con 0x7f4be006b920 2023-10-12T22:38:34.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.039+0000 7f4bd5ffb700 1 --2- 172.21.15.195:0/629141772 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4bcc02ff60 0x7f4bcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:34.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.039+0000 7f4bd5ffb700 1 -- 172.21.15.195:0/629141772 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4bdc040a20 con 0x7f4be006b920 2023-10-12T22:38:34.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.039+0000 7f4be631b700 1 -- 172.21.15.195:0/629141772 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be0045ad0 con 0x7f4be006b920 2023-10-12T22:38:34.044 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.039+0000 7f4be4b18700 1 --2- 172.21.15.195:0/629141772 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4bcc02ff60 0x7f4bcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:34.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.040+0000 7f4be4b18700 1 --2- 172.21.15.195:0/629141772 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4bcc02ff60 0x7f4bcc032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f4bd00096a0 tx=0x7f4bd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:34.045 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.044+0000 7f4bd5ffb700 1 -- 172.21.15.195:0/629141772 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4bdc026350 con 0x7f4be006b920 2023-10-12T22:38:34.250 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.249+0000 7f4be631b700 1 -- 172.21.15.195:0/629141772 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4be00f9ac0 con 0x7f4be006b920 2023-10-12T22:38:34.251 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.250+0000 7f4bd5ffb700 1 -- 172.21.15.195:0/629141772 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4bdc012330 con 0x7f4be006b920 2023-10-12T22:38:34.251 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:34.252 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:34.253 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.251+0000 7f4be631b700 1 -- 172.21.15.195:0/629141772 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4bcc02ff60 msgr2=0x7f4bcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:34.253 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.251+0000 7f4be631b700 1 --2- 172.21.15.195:0/629141772 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4bcc02ff60 0x7f4bcc032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f4bd00096a0 tx=0x7f4bd0006b40).stop 2023-10-12T22:38:34.253 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.251+0000 7f4be631b700 1 -- 172.21.15.195:0/629141772 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4be006b920 msgr2=0x7f4be010cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:34.253 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.251+0000 7f4be631b700 1 --2- 172.21.15.195:0/629141772 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4be006b920 0x7f4be010cac0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f4bdc014000 tx=0x7f4bdc00b040).stop 2023-10-12T22:38:34.253 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.251+0000 7f4be631b700 1 -- 172.21.15.195:0/629141772 shutdown_connections 2023-10-12T22:38:34.254 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.251+0000 7f4be631b700 1 --2- 172.21.15.195:0/629141772 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4bcc02ff60 0x7f4bcc032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:34.254 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.251+0000 7f4be631b700 1 --2- 172.21.15.195:0/629141772 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4be006b920 0x7f4be010cac0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:34.254 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.251+0000 7f4be631b700 1 -- 172.21.15.195:0/629141772 >> 172.21.15.195:0/629141772 conn(0x7f4be00f1ac0 msgr2=0x7f4be00f25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:34.254 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.251+0000 7f4be631b700 1 -- 172.21.15.195:0/629141772 shutdown_connections 2023-10-12T22:38:34.254 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:34.251+0000 7f4be631b700 1 -- 172.21.15.195:0/629141772 wait complete. 2023-10-12T22:38:34.255 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:34.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:32.864065+0000 mgr.smithi131.efktma (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:35.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:35 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:34.251294+0000 mon.smithi131 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.195:0/629141772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:35.643 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:35.644 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:36.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:36 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:34.864446+0000 mgr.smithi131.efktma (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:36.741 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.739+0000 7feedb841700 1 -- 172.21.15.195:0/881614177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feed4060280 msgr2=0x7feed40f90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:36.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.739+0000 7feedb841700 1 --2- 172.21.15.195:0/881614177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feed4060280 0x7feed40f90d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7feec4009fb0 tx=0x7feec4009b30).stop 2023-10-12T22:38:36.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.739+0000 7feedb841700 1 -- 172.21.15.195:0/881614177 shutdown_connections 2023-10-12T22:38:36.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.739+0000 7feedb841700 1 --2- 172.21.15.195:0/881614177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feed4060280 0x7feed40f90d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:36.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.739+0000 7feedb841700 1 -- 172.21.15.195:0/881614177 >> 172.21.15.195:0/881614177 conn(0x7feed40f0b70 msgr2=0x7feed40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:36.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.739+0000 7feedb841700 1 -- 172.21.15.195:0/881614177 shutdown_connections 2023-10-12T22:38:36.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.739+0000 7feedb841700 1 -- 172.21.15.195:0/881614177 wait complete. 2023-10-12T22:38:36.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.740+0000 7feedb841700 1 Processor -- start 2023-10-12T22:38:36.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.740+0000 7feedb841700 1 -- start start 2023-10-12T22:38:36.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.740+0000 7feedb841700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feed4060280 0x7feed41066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:36.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.740+0000 7feedb841700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed4106bc0 con 0x7feed4060280 2023-10-12T22:38:36.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.740+0000 7feed95dd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feed4060280 0x7feed41066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:36.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.740+0000 7feed95dd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feed4060280 0x7feed41066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:55728/0 (socket says 172.21.15.195:55728) 2023-10-12T22:38:36.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.740+0000 7feed95dd700 1 -- 172.21.15.195:0/1568011224 learned_addr learned my addr 172.21.15.195:0/1568011224 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:36.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.741+0000 7feed95dd700 1 -- 172.21.15.195:0/1568011224 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec4005040 con 0x7feed4060280 2023-10-12T22:38:36.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.741+0000 7feed95dd700 1 --2- 172.21.15.195:0/1568011224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feed4060280 0x7feed41066d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7feec4013510 tx=0x7feec400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:36.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.741+0000 7feec9ffb700 1 -- 172.21.15.195:0/1568011224 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec4009320 con 0x7feed4060280 2023-10-12T22:38:36.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.741+0000 7feedb841700 1 -- 172.21.15.195:0/1568011224 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed4106dc0 con 0x7feed4060280 2023-10-12T22:38:36.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.741+0000 7feec9ffb700 1 -- 172.21.15.195:0/1568011224 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feec4013f00 con 0x7feed4060280 2023-10-12T22:38:36.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.741+0000 7feec9ffb700 1 -- 172.21.15.195:0/1568011224 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec4014dc0 con 0x7feed4060280 2023-10-12T22:38:36.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.741+0000 7feedb841700 1 -- 172.21.15.195:0/1568011224 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed4107250 con 0x7feed4060280 2023-10-12T22:38:36.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.742+0000 7feec9ffb700 1 -- 172.21.15.195:0/1568011224 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feec4014420 con 0x7feed4060280 2023-10-12T22:38:36.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.742+0000 7feedb841700 1 -- 172.21.15.195:0/1568011224 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed4045ad0 con 0x7feed4060280 2023-10-12T22:38:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.742+0000 7feec9ffb700 1 --2- 172.21.15.195:0/1568011224 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7feec002ff60 0x7feec0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.742+0000 7feec9ffb700 1 -- 172.21.15.195:0/1568011224 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feec40410f0 con 0x7feed4060280 2023-10-12T22:38:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.742+0000 7feed8ddc700 1 --2- 172.21.15.195:0/1568011224 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7feec002ff60 0x7feec0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:36.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.743+0000 7feed8ddc700 1 --2- 172.21.15.195:0/1568011224 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7feec002ff60 0x7feec0032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7feed00096a0 tx=0x7feed0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:36.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.747+0000 7feec9ffb700 1 -- 172.21.15.195:0/1568011224 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feec40095f0 con 0x7feed4060280 2023-10-12T22:38:36.954 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.953+0000 7feedb841700 1 -- 172.21.15.195:0/1568011224 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feed4108100 con 0x7feed4060280 2023-10-12T22:38:36.955 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.954+0000 7feec9ffb700 1 -- 172.21.15.195:0/1568011224 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feec401cb10 con 0x7feed4060280 2023-10-12T22:38:36.955 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:36.956 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:36.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.955+0000 7feedb841700 1 -- 172.21.15.195:0/1568011224 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7feec002ff60 msgr2=0x7feec0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:36.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.955+0000 7feedb841700 1 --2- 172.21.15.195:0/1568011224 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7feec002ff60 0x7feec0032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7feed00096a0 tx=0x7feed0006b40).stop 2023-10-12T22:38:36.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.955+0000 7feedb841700 1 -- 172.21.15.195:0/1568011224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feed4060280 msgr2=0x7feed41066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:36.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.955+0000 7feedb841700 1 --2- 172.21.15.195:0/1568011224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feed4060280 0x7feed41066d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7feec4013510 tx=0x7feec400b040).stop 2023-10-12T22:38:36.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.955+0000 7feedb841700 1 -- 172.21.15.195:0/1568011224 shutdown_connections 2023-10-12T22:38:36.958 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.955+0000 7feedb841700 1 --2- 172.21.15.195:0/1568011224 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7feec002ff60 0x7feec0032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:36.958 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.955+0000 7feedb841700 1 --2- 172.21.15.195:0/1568011224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feed4060280 0x7feed41066d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:36.958 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.955+0000 7feedb841700 1 -- 172.21.15.195:0/1568011224 >> 172.21.15.195:0/1568011224 conn(0x7feed40f0b70 msgr2=0x7feed40f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:36.958 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.956+0000 7feedb841700 1 -- 172.21.15.195:0/1568011224 shutdown_connections 2023-10-12T22:38:36.959 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:36.956+0000 7feedb841700 1 -- 172.21.15.195:0/1568011224 wait complete. 2023-10-12T22:38:36.959 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:37.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:37 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:36.955153+0000 mon.smithi131 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.195:0/1568011224' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:38.340 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:38.340 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:38.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:38 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:36.864785+0000 mgr.smithi131.efktma (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:39.498 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.495+0000 7f2e7424d700 1 -- 172.21.15.195:0/2733674732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c0f6510 msgr2=0x7f2e6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:39.498 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.495+0000 7f2e7424d700 1 --2- 172.21.15.195:0/2733674732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c0f6510 0x7f2e6c0f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f2e68009fb0 tx=0x7f2e68009b30).stop 2023-10-12T22:38:39.498 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.495+0000 7f2e7424d700 1 -- 172.21.15.195:0/2733674732 shutdown_connections 2023-10-12T22:38:39.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.495+0000 7f2e7424d700 1 --2- 172.21.15.195:0/2733674732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c0f6510 0x7f2e6c0f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:39.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.495+0000 7f2e7424d700 1 -- 172.21.15.195:0/2733674732 >> 172.21.15.195:0/2733674732 conn(0x7f2e6c0f1c70 msgr2=0x7f2e6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:39.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.495+0000 7f2e7424d700 1 -- 172.21.15.195:0/2733674732 shutdown_connections 2023-10-12T22:38:39.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.495+0000 7f2e7424d700 1 -- 172.21.15.195:0/2733674732 wait complete. 2023-10-12T22:38:39.499 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.496+0000 7f2e7424d700 1 Processor -- start 2023-10-12T22:38:39.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.496+0000 7f2e7424d700 1 -- start start 2023-10-12T22:38:39.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.496+0000 7f2e7424d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c0f6510 0x7f2e6c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:39.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.496+0000 7f2e7424d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c069e20 con 0x7f2e6c0f6510 2023-10-12T22:38:39.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.496+0000 7f2e71fe9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c0f6510 0x7f2e6c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:39.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.496+0000 7f2e71fe9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c0f6510 0x7f2e6c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:43396/0 (socket says 172.21.15.195:43396) 2023-10-12T22:38:39.500 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.496+0000 7f2e71fe9700 1 -- 172.21.15.195:0/3493654876 learned_addr learned my addr 172.21.15.195:0/3493654876 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:39.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.496+0000 7f2e71fe9700 1 -- 172.21.15.195:0/3493654876 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68005040 con 0x7f2e6c0f6510 2023-10-12T22:38:39.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.497+0000 7f2e71fe9700 1 --2- 172.21.15.195:0/3493654876 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c0f6510 0x7f2e6c06b9a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f2e68013510 tx=0x7f2e6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:39.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.497+0000 7f2e627fc700 1 -- 172.21.15.195:0/3493654876 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e68009320 con 0x7f2e6c0f6510 2023-10-12T22:38:39.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.497+0000 7f2e7424d700 1 -- 172.21.15.195:0/3493654876 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6c069fc0 con 0x7f2e6c0f6510 2023-10-12T22:38:39.501 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.497+0000 7f2e627fc700 1 -- 172.21.15.195:0/3493654876 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e68013f00 con 0x7f2e6c0f6510 2023-10-12T22:38:39.502 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.497+0000 7f2e627fc700 1 -- 172.21.15.195:0/3493654876 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e68014d90 con 0x7f2e6c0f6510 2023-10-12T22:38:39.502 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.497+0000 7f2e7424d700 1 -- 172.21.15.195:0/3493654876 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6c06a450 con 0x7f2e6c0f6510 2023-10-12T22:38:39.503 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.497+0000 7f2e627fc700 1 -- 172.21.15.195:0/3493654876 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e680263a0 con 0x7f2e6c0f6510 2023-10-12T22:38:39.503 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.498+0000 7f2e627fc700 1 --2- 172.21.15.195:0/3493654876 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2e5802ff60 0x7f2e58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:39.503 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.498+0000 7f2e627fc700 1 -- 172.21.15.195:0/3493654876 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e680410b0 con 0x7f2e6c0f6510 2023-10-12T22:38:39.503 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.498+0000 7f2e7424d700 1 -- 172.21.15.195:0/3493654876 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6c059100 con 0x7f2e6c0f6510 2023-10-12T22:38:39.504 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.498+0000 7f2e717e8700 1 --2- 172.21.15.195:0/3493654876 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2e5802ff60 0x7f2e58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:39.504 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.498+0000 7f2e717e8700 1 --2- 172.21.15.195:0/3493654876 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2e5802ff60 0x7f2e58032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f2e5c0096a0 tx=0x7f2e5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:39.504 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.503+0000 7f2e627fc700 1 -- 172.21.15.195:0/3493654876 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e68012920 con 0x7f2e6c0f6510 2023-10-12T22:38:39.711 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.710+0000 7f2e7424d700 1 -- 172.21.15.195:0/3493654876 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e6c06aab0 con 0x7f2e6c0f6510 2023-10-12T22:38:39.712 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.711+0000 7f2e627fc700 1 -- 172.21.15.195:0/3493654876 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2e6802b020 con 0x7f2e6c0f6510 2023-10-12T22:38:39.713 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:39.713 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:39.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.712+0000 7f2e7424d700 1 -- 172.21.15.195:0/3493654876 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2e5802ff60 msgr2=0x7f2e58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:39.714 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.712+0000 7f2e7424d700 1 --2- 172.21.15.195:0/3493654876 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2e5802ff60 0x7f2e58032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f2e5c0096a0 tx=0x7f2e5c006b40).stop 2023-10-12T22:38:39.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.712+0000 7f2e7424d700 1 -- 172.21.15.195:0/3493654876 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c0f6510 msgr2=0x7f2e6c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:39.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.712+0000 7f2e7424d700 1 --2- 172.21.15.195:0/3493654876 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c0f6510 0x7f2e6c06b9a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f2e68013510 tx=0x7f2e6800b040).stop 2023-10-12T22:38:39.715 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.712+0000 7f2e7424d700 1 -- 172.21.15.195:0/3493654876 shutdown_connections 2023-10-12T22:38:39.716 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.712+0000 7f2e7424d700 1 --2- 172.21.15.195:0/3493654876 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2e5802ff60 0x7f2e58032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:39.716 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.712+0000 7f2e7424d700 1 --2- 172.21.15.195:0/3493654876 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e6c0f6510 0x7f2e6c06b9a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:39.716 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.712+0000 7f2e7424d700 1 -- 172.21.15.195:0/3493654876 >> 172.21.15.195:0/3493654876 conn(0x7f2e6c0f1c70 msgr2=0x7f2e6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:39.716 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.712+0000 7f2e7424d700 1 -- 172.21.15.195:0/3493654876 shutdown_connections 2023-10-12T22:38:39.717 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:39.712+0000 7f2e7424d700 1 -- 172.21.15.195:0/3493654876 wait complete. 2023-10-12T22:38:39.717 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:40.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:40 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:38.865115+0000 mgr.smithi131.efktma (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:40.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:40 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:39.712286+0000 mon.smithi131 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.195:0/3493654876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:41.088 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:41.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.217+0000 7f48ab535700 1 -- 172.21.15.195:0/3999436392 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a40f4740 msgr2=0x7f48a40f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.217+0000 7f48ab535700 1 --2- 172.21.15.195:0/3999436392 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a40f4740 0x7f48a40f4b60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f4894009fb0 tx=0x7f4894009b30).stop 2023-10-12T22:38:42.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.217+0000 7f48ab535700 1 -- 172.21.15.195:0/3999436392 shutdown_connections 2023-10-12T22:38:42.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.217+0000 7f48ab535700 1 --2- 172.21.15.195:0/3999436392 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a40f4740 0x7f48a40f4b60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:42.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.217+0000 7f48ab535700 1 -- 172.21.15.195:0/3999436392 >> 172.21.15.195:0/3999436392 conn(0x7f48a40f0260 msgr2=0x7f48a40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:42.221 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.217+0000 7f48ab535700 1 -- 172.21.15.195:0/3999436392 shutdown_connections 2023-10-12T22:38:42.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.217+0000 7f48ab535700 1 -- 172.21.15.195:0/3999436392 wait complete. 2023-10-12T22:38:42.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.218+0000 7f48ab535700 1 Processor -- start 2023-10-12T22:38:42.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.218+0000 7f48ab535700 1 -- start start 2023-10-12T22:38:42.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.218+0000 7f48ab535700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a40f4740 0x7f48a4106690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:42.222 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.218+0000 7f48ab535700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a4106b80 con 0x7f48a40f4740 2023-10-12T22:38:42.223 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.218+0000 7f48a92d1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a40f4740 0x7f48a4106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:42.223 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.218+0000 7f48a92d1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a40f4740 0x7f48a4106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:43412/0 (socket says 172.21.15.195:43412) 2023-10-12T22:38:42.223 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.218+0000 7f48a92d1700 1 -- 172.21.15.195:0/4104042852 learned_addr learned my addr 172.21.15.195:0/4104042852 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:42.223 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.219+0000 7f48a92d1700 1 -- 172.21.15.195:0/4104042852 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4894005040 con 0x7f48a40f4740 2023-10-12T22:38:42.224 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.219+0000 7f48a92d1700 1 --2- 172.21.15.195:0/4104042852 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a40f4740 0x7f48a4106690 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f4894014000 tx=0x7f489400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:42.224 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.219+0000 7f4899ffb700 1 -- 172.21.15.195:0/4104042852 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48940095f0 con 0x7f48a40f4740 2023-10-12T22:38:42.224 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.219+0000 7f4899ffb700 1 -- 172.21.15.195:0/4104042852 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4894007e70 con 0x7f48a40f4740 2023-10-12T22:38:42.224 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.219+0000 7f48ab535700 1 -- 172.21.15.195:0/4104042852 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a4106d80 con 0x7f48a40f4740 2023-10-12T22:38:42.224 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.219+0000 7f4899ffb700 1 -- 172.21.15.195:0/4104042852 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4894014ea0 con 0x7f48a40f4740 2023-10-12T22:38:42.225 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.219+0000 7f48ab535700 1 -- 172.21.15.195:0/4104042852 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a4107210 con 0x7f48a40f4740 2023-10-12T22:38:42.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.220+0000 7f4899ffb700 1 -- 172.21.15.195:0/4104042852 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4894025070 con 0x7f48a40f4740 2023-10-12T22:38:42.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.220+0000 7f48ab535700 1 -- 172.21.15.195:0/4104042852 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a4045ad0 con 0x7f48a40f4740 2023-10-12T22:38:42.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.220+0000 7f4899ffb700 1 --2- 172.21.15.195:0/4104042852 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f489002ff60 0x7f4890032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:42.226 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.220+0000 7f4899ffb700 1 -- 172.21.15.195:0/4104042852 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4894040a00 con 0x7f48a40f4740 2023-10-12T22:38:42.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.220+0000 7f48a8ad0700 1 --2- 172.21.15.195:0/4104042852 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f489002ff60 0x7f4890032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:42.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.221+0000 7f48a8ad0700 1 --2- 172.21.15.195:0/4104042852 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f489002ff60 0x7f4890032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f48a00096a0 tx=0x7f48a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:42.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.225+0000 7f4899ffb700 1 -- 172.21.15.195:0/4104042852 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48940079c0 con 0x7f48a40f4740 2023-10-12T22:38:42.429 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.428+0000 7f48ab535700 1 -- 172.21.15.195:0/4104042852 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48a4108060 con 0x7f48a40f4740 2023-10-12T22:38:42.430 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.429+0000 7f4899ffb700 1 -- 172.21.15.195:0/4104042852 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f489400f300 con 0x7f48a40f4740 2023-10-12T22:38:42.430 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:42.431 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:42.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.430+0000 7f48ab535700 1 -- 172.21.15.195:0/4104042852 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f489002ff60 msgr2=0x7f4890032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:42.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.430+0000 7f48ab535700 1 --2- 172.21.15.195:0/4104042852 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f489002ff60 0x7f4890032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f48a00096a0 tx=0x7f48a0006b40).stop 2023-10-12T22:38:42.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.430+0000 7f48ab535700 1 -- 172.21.15.195:0/4104042852 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a40f4740 msgr2=0x7f48a4106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:42.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.430+0000 7f48ab535700 1 --2- 172.21.15.195:0/4104042852 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a40f4740 0x7f48a4106690 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f4894014000 tx=0x7f489400b040).stop 2023-10-12T22:38:42.433 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.430+0000 7f48ab535700 1 -- 172.21.15.195:0/4104042852 shutdown_connections 2023-10-12T22:38:42.433 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.430+0000 7f48ab535700 1 --2- 172.21.15.195:0/4104042852 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f489002ff60 0x7f4890032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:42.433 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.430+0000 7f48ab535700 1 --2- 172.21.15.195:0/4104042852 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48a40f4740 0x7f48a4106690 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:42.433 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.430+0000 7f48ab535700 1 -- 172.21.15.195:0/4104042852 >> 172.21.15.195:0/4104042852 conn(0x7f48a40f0260 msgr2=0x7f48a40f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:42.433 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.431+0000 7f48ab535700 1 -- 172.21.15.195:0/4104042852 shutdown_connections 2023-10-12T22:38:42.434 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:42.431+0000 7f48ab535700 1 -- 172.21.15.195:0/4104042852 wait complete. 2023-10-12T22:38:42.434 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:42.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:42 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:40.865479+0000 mgr.smithi131.efktma (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:43.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:43 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:42.430387+0000 mon.smithi131 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.195:0/4104042852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:43.812 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:43.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:44.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:44 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:42.865769+0000 mgr.smithi131.efktma (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:44.971 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.969+0000 7fe0e04b9700 1 -- 172.21.15.195:0/2725532474 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0d80f6510 msgr2=0x7fe0d80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:44.971 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.969+0000 7fe0e04b9700 1 --2- 172.21.15.195:0/2725532474 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0d80f6510 0x7fe0d80f6930 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fe0d0009960 tx=0x7fe0d0004d10).stop 2023-10-12T22:38:44.971 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.969+0000 7fe0e04b9700 1 -- 172.21.15.195:0/2725532474 shutdown_connections 2023-10-12T22:38:44.971 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.969+0000 7fe0e04b9700 1 --2- 172.21.15.195:0/2725532474 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0d80f6510 0x7fe0d80f6930 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:44.971 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.969+0000 7fe0e04b9700 1 -- 172.21.15.195:0/2725532474 >> 172.21.15.195:0/2725532474 conn(0x7fe0d80f1c70 msgr2=0x7fe0d80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:44.972 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.969+0000 7fe0e04b9700 1 -- 172.21.15.195:0/2725532474 shutdown_connections 2023-10-12T22:38:44.972 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.969+0000 7fe0e04b9700 1 -- 172.21.15.195:0/2725532474 wait complete. 2023-10-12T22:38:44.972 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.970+0000 7fe0e04b9700 1 Processor -- start 2023-10-12T22:38:44.972 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.970+0000 7fe0e04b9700 1 -- start start 2023-10-12T22:38:44.974 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.970+0000 7fe0e04b9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0d80f6510 0x7fe0d8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:44.974 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.970+0000 7fe0e04b9700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d8106c20 con 0x7fe0d80f6510 2023-10-12T22:38:44.974 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.970+0000 7fe0de255700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0d80f6510 0x7fe0d8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:44.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.970+0000 7fe0de255700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0d80f6510 0x7fe0d8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:43434/0 (socket says 172.21.15.195:43434) 2023-10-12T22:38:44.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.970+0000 7fe0de255700 1 -- 172.21.15.195:0/1366151980 learned_addr learned my addr 172.21.15.195:0/1366151980 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:44.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.971+0000 7fe0de255700 1 -- 172.21.15.195:0/1366151980 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d0009640 con 0x7fe0d80f6510 2023-10-12T22:38:44.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.971+0000 7fe0de255700 1 --2- 172.21.15.195:0/1366151980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0d80f6510 0x7fe0d8106730 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fe0d0004b70 tx=0x7fe0d0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:44.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.971+0000 7fe0d6ffd700 1 -- 172.21.15.195:0/1366151980 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0d0018070 con 0x7fe0d80f6510 2023-10-12T22:38:44.976 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.971+0000 7fe0e04b9700 1 -- 172.21.15.195:0/1366151980 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d8106e20 con 0x7fe0d80f6510 2023-10-12T22:38:44.976 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.971+0000 7fe0d6ffd700 1 -- 172.21.15.195:0/1366151980 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0d000b870 con 0x7fe0d80f6510 2023-10-12T22:38:44.976 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.971+0000 7fe0d6ffd700 1 -- 172.21.15.195:0/1366151980 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0d001c560 con 0x7fe0d80f6510 2023-10-12T22:38:44.976 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.971+0000 7fe0e04b9700 1 -- 172.21.15.195:0/1366151980 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d81072b0 con 0x7fe0d80f6510 2023-10-12T22:38:44.976 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.972+0000 7fe0d6ffd700 1 -- 172.21.15.195:0/1366151980 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0d001c6c0 con 0x7fe0d80f6510 2023-10-12T22:38:44.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.972+0000 7fe0d6ffd700 1 --2- 172.21.15.195:0/1366151980 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe0c002ff60 0x7fe0c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:44.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.972+0000 7fe0d6ffd700 1 -- 172.21.15.195:0/1366151980 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0d00412a0 con 0x7fe0d80f6510 2023-10-12T22:38:44.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.972+0000 7fe0e04b9700 1 -- 172.21.15.195:0/1366151980 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d8045ad0 con 0x7fe0d80f6510 2023-10-12T22:38:44.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.972+0000 7fe0dda54700 1 --2- 172.21.15.195:0/1366151980 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe0c002ff60 0x7fe0c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:44.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.972+0000 7fe0dda54700 1 --2- 172.21.15.195:0/1366151980 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe0c002ff60 0x7fe0c0032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe0cc0096a0 tx=0x7fe0cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:44.978 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:44.976+0000 7fe0d6ffd700 1 -- 172.21.15.195:0/1366151980 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0d00127f0 con 0x7fe0d80f6510 2023-10-12T22:38:45.184 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:45.182+0000 7fe0e04b9700 1 -- 172.21.15.195:0/1366151980 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0d80fc050 con 0x7fe0d80f6510 2023-10-12T22:38:45.184 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:45.183+0000 7fe0d6ffd700 1 -- 172.21.15.195:0/1366151980 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe0d0019230 con 0x7fe0d80f6510 2023-10-12T22:38:45.185 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:45.185 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:45.186 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:45.184+0000 7fe0e04b9700 1 -- 172.21.15.195:0/1366151980 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe0c002ff60 msgr2=0x7fe0c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:45.186 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:45.184+0000 7fe0e04b9700 1 --2- 172.21.15.195:0/1366151980 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe0c002ff60 0x7fe0c0032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe0cc0096a0 tx=0x7fe0cc006b40).stop 2023-10-12T22:38:45.187 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:45.184+0000 7fe0e04b9700 1 -- 172.21.15.195:0/1366151980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0d80f6510 msgr2=0x7fe0d8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:45.187 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:45.184+0000 7fe0e04b9700 1 --2- 172.21.15.195:0/1366151980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0d80f6510 0x7fe0d8106730 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fe0d0004b70 tx=0x7fe0d0004d10).stop 2023-10-12T22:38:45.187 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:45.184+0000 7fe0e04b9700 1 -- 172.21.15.195:0/1366151980 shutdown_connections 2023-10-12T22:38:45.187 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:45.184+0000 7fe0e04b9700 1 --2- 172.21.15.195:0/1366151980 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe0c002ff60 0x7fe0c0032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:45.187 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:45.184+0000 7fe0e04b9700 1 --2- 172.21.15.195:0/1366151980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe0d80f6510 0x7fe0d8106730 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:45.188 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:45.184+0000 7fe0e04b9700 1 -- 172.21.15.195:0/1366151980 >> 172.21.15.195:0/1366151980 conn(0x7fe0d80f1c70 msgr2=0x7fe0d80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:45.188 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:45.185+0000 7fe0e04b9700 1 -- 172.21.15.195:0/1366151980 shutdown_connections 2023-10-12T22:38:45.188 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:45.185+0000 7fe0e04b9700 1 -- 172.21.15.195:0/1366151980 wait complete. 2023-10-12T22:38:45.188 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:46.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:46 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:44.866136+0000 mgr.smithi131.efktma (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:46.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:46 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:45.184480+0000 mon.smithi131 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.195:0/1366151980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:46.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:46.565 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:47.702 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.701+0000 7fccd7fff700 1 -- 172.21.15.195:0/995979641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd80f51a0 msgr2=0x7fccd80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:47.702 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.701+0000 7fccd7fff700 1 --2- 172.21.15.195:0/995979641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd80f51a0 0x7fccd80f55c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fccc8009fb0 tx=0x7fccc8009b30).stop 2023-10-12T22:38:47.703 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.701+0000 7fccd7fff700 1 -- 172.21.15.195:0/995979641 shutdown_connections 2023-10-12T22:38:47.703 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.701+0000 7fccd7fff700 1 --2- 172.21.15.195:0/995979641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd80f51a0 0x7fccd80f55c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:47.703 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.701+0000 7fccd7fff700 1 -- 172.21.15.195:0/995979641 >> 172.21.15.195:0/995979641 conn(0x7fccd80f09a0 msgr2=0x7fccd80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:47.703 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.701+0000 7fccd7fff700 1 -- 172.21.15.195:0/995979641 shutdown_connections 2023-10-12T22:38:47.703 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.701+0000 7fccd7fff700 1 -- 172.21.15.195:0/995979641 wait complete. 2023-10-12T22:38:47.705 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.701+0000 7fccd7fff700 1 Processor -- start 2023-10-12T22:38:47.706 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.701+0000 7fccd7fff700 1 -- start start 2023-10-12T22:38:47.706 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.702+0000 7fccd7fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd80f51a0 0x7fccd81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:47.706 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.702+0000 7fccd7fff700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd81058a0 con 0x7fccd80f51a0 2023-10-12T22:38:47.706 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.702+0000 7fccd6ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd80f51a0 0x7fccd81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:47.706 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.702+0000 7fccd6ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd80f51a0 0x7fccd81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:43448/0 (socket says 172.21.15.195:43448) 2023-10-12T22:38:47.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.702+0000 7fccd6ffd700 1 -- 172.21.15.195:0/484987680 learned_addr learned my addr 172.21.15.195:0/484987680 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:47.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.702+0000 7fccd6ffd700 1 -- 172.21.15.195:0/484987680 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc8005040 con 0x7fccd80f51a0 2023-10-12T22:38:47.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.702+0000 7fccd6ffd700 1 --2- 172.21.15.195:0/484987680 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd80f51a0 0x7fccd81053b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fccc8013510 tx=0x7fccc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:47.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.702+0000 7fccbf7fe700 1 -- 172.21.15.195:0/484987680 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccc8009320 con 0x7fccd80f51a0 2023-10-12T22:38:47.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.702+0000 7fccd7fff700 1 -- 172.21.15.195:0/484987680 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd8105aa0 con 0x7fccd80f51a0 2023-10-12T22:38:47.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.702+0000 7fccbf7fe700 1 -- 172.21.15.195:0/484987680 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccc8013f00 con 0x7fccd80f51a0 2023-10-12T22:38:47.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.703+0000 7fccbf7fe700 1 -- 172.21.15.195:0/484987680 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccc8014dc0 con 0x7fccd80f51a0 2023-10-12T22:38:47.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.703+0000 7fccd7fff700 1 -- 172.21.15.195:0/484987680 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd8105f30 con 0x7fccd80f51a0 2023-10-12T22:38:47.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.703+0000 7fccbf7fe700 1 -- 172.21.15.195:0/484987680 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fccc80095f0 con 0x7fccd80f51a0 2023-10-12T22:38:47.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.703+0000 7fccd7fff700 1 -- 172.21.15.195:0/484987680 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd8045ad0 con 0x7fccd80f51a0 2023-10-12T22:38:47.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.704+0000 7fccbf7fe700 1 --2- 172.21.15.195:0/484987680 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fccc002ff60 0x7fccc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:47.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.704+0000 7fccbf7fe700 1 -- 172.21.15.195:0/484987680 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fccc8041000 con 0x7fccd80f51a0 2023-10-12T22:38:47.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.704+0000 7fccd67fc700 1 --2- 172.21.15.195:0/484987680 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fccc002ff60 0x7fccc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:47.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.704+0000 7fccd67fc700 1 --2- 172.21.15.195:0/484987680 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fccc002ff60 0x7fccc0032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fcccc0096a0 tx=0x7fcccc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:47.710 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.708+0000 7fccbf7fe700 1 -- 172.21.15.195:0/484987680 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fccc8014420 con 0x7fccd80f51a0 2023-10-12T22:38:47.928 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.917+0000 7fccd7fff700 1 -- 172.21.15.195:0/484987680 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccd8106de0 con 0x7fccd80f51a0 2023-10-12T22:38:47.929 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.918+0000 7fccbf7fe700 1 -- 172.21.15.195:0/484987680 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fccc8010230 con 0x7fccd80f51a0 2023-10-12T22:38:47.929 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:47.929 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:47.930 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.919+0000 7fccd7fff700 1 -- 172.21.15.195:0/484987680 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fccc002ff60 msgr2=0x7fccc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:47.930 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.919+0000 7fccd7fff700 1 --2- 172.21.15.195:0/484987680 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fccc002ff60 0x7fccc0032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fcccc0096a0 tx=0x7fcccc006b40).stop 2023-10-12T22:38:47.931 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.920+0000 7fccd7fff700 1 -- 172.21.15.195:0/484987680 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd80f51a0 msgr2=0x7fccd81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:47.931 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.920+0000 7fccd7fff700 1 --2- 172.21.15.195:0/484987680 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd80f51a0 0x7fccd81053b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fccc8013510 tx=0x7fccc800b040).stop 2023-10-12T22:38:47.931 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.920+0000 7fccd7fff700 1 -- 172.21.15.195:0/484987680 shutdown_connections 2023-10-12T22:38:47.931 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.920+0000 7fccd7fff700 1 --2- 172.21.15.195:0/484987680 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fccc002ff60 0x7fccc0032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:47.931 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.920+0000 7fccd7fff700 1 --2- 172.21.15.195:0/484987680 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd80f51a0 0x7fccd81053b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:47.932 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.920+0000 7fccd7fff700 1 -- 172.21.15.195:0/484987680 >> 172.21.15.195:0/484987680 conn(0x7fccd80f09a0 msgr2=0x7fccd80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:47.932 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.920+0000 7fccd7fff700 1 -- 172.21.15.195:0/484987680 shutdown_connections 2023-10-12T22:38:47.932 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:47.920+0000 7fccd7fff700 1 -- 172.21.15.195:0/484987680 wait complete. 2023-10-12T22:38:47.932 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:48.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:48 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:46.866485+0000 mgr.smithi131.efktma (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:48.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:48 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:47.919490+0000 mon.smithi131 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.195:0/484987680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:49.298 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:49.298 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:50.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.478+0000 7f67b9d9b700 1 -- 172.21.15.195:0/3748196655 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67a40138a0 con 0x7f67bc0f53f0 2023-10-12T22:38:50.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.479+0000 7f67c118d700 1 -- 172.21.15.195:0/3748196655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67bc0f53f0 msgr2=0x7f67bc0f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:50.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.479+0000 7f67c118d700 1 --2- 172.21.15.195:0/3748196655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67bc0f53f0 0x7f67bc0f5810 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f67a4009fb0 tx=0x7f67a4009b30).stop 2023-10-12T22:38:50.482 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.479+0000 7f67c118d700 1 -- 172.21.15.195:0/3748196655 shutdown_connections 2023-10-12T22:38:50.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.479+0000 7f67c118d700 1 --2- 172.21.15.195:0/3748196655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67bc0f53f0 0x7f67bc0f5810 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:50.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.479+0000 7f67c118d700 1 -- 172.21.15.195:0/3748196655 >> 172.21.15.195:0/3748196655 conn(0x7f67bc0f0b70 msgr2=0x7f67bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:50.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.479+0000 7f67c118d700 1 -- 172.21.15.195:0/3748196655 shutdown_connections 2023-10-12T22:38:50.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.479+0000 7f67c118d700 1 -- 172.21.15.195:0/3748196655 wait complete. 2023-10-12T22:38:50.483 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.480+0000 7f67c118d700 1 Processor -- start 2023-10-12T22:38:50.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.480+0000 7f67c118d700 1 -- start start 2023-10-12T22:38:50.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.480+0000 7f67c118d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67bc0f53f0 0x7f67bc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:50.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.480+0000 7f67c118d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc106bd0 con 0x7f67bc0f53f0 2023-10-12T22:38:50.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.480+0000 7f67bad9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67bc0f53f0 0x7f67bc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:50.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.480+0000 7f67bad9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67bc0f53f0 0x7f67bc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:53318/0 (socket says 172.21.15.195:53318) 2023-10-12T22:38:50.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.480+0000 7f67bad9d700 1 -- 172.21.15.195:0/3378305941 learned_addr learned my addr 172.21.15.195:0/3378305941 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:50.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.480+0000 7f67bad9d700 1 -- 172.21.15.195:0/3378305941 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a4005040 con 0x7f67bc0f53f0 2023-10-12T22:38:50.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.481+0000 7f67bad9d700 1 --2- 172.21.15.195:0/3378305941 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67bc0f53f0 0x7f67bc1066e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f67a4013510 tx=0x7f67a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:50.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.481+0000 7f67b37fe700 1 -- 172.21.15.195:0/3378305941 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67a4009320 con 0x7f67bc0f53f0 2023-10-12T22:38:50.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.481+0000 7f67c118d700 1 -- 172.21.15.195:0/3378305941 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67bc106dd0 con 0x7f67bc0f53f0 2023-10-12T22:38:50.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.481+0000 7f67b37fe700 1 -- 172.21.15.195:0/3378305941 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67a4013f00 con 0x7f67bc0f53f0 2023-10-12T22:38:50.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.481+0000 7f67b37fe700 1 -- 172.21.15.195:0/3378305941 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67a4014d90 con 0x7f67bc0f53f0 2023-10-12T22:38:50.486 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.481+0000 7f67c118d700 1 -- 172.21.15.195:0/3378305941 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67bc107260 con 0x7f67bc0f53f0 2023-10-12T22:38:50.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.481+0000 7f67b37fe700 1 -- 172.21.15.195:0/3378305941 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67a40095f0 con 0x7f67bc0f53f0 2023-10-12T22:38:50.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.482+0000 7f67b37fe700 1 --2- 172.21.15.195:0/3378305941 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f67a802ff60 0x7f67a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:50.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.482+0000 7f67c118d700 1 -- 172.21.15.195:0/3378305941 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67bc045ad0 con 0x7f67bc0f53f0 2023-10-12T22:38:50.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.482+0000 7f67b37fe700 1 -- 172.21.15.195:0/3378305941 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67a40410c0 con 0x7f67bc0f53f0 2023-10-12T22:38:50.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.482+0000 7f67ba59c700 1 --2- 172.21.15.195:0/3378305941 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f67a802ff60 0x7f67a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:50.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.482+0000 7f67ba59c700 1 --2- 172.21.15.195:0/3378305941 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f67a802ff60 0x7f67a8032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f67ac0096a0 tx=0x7f67ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:50.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.486+0000 7f67b37fe700 1 -- 172.21.15.195:0/3378305941 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67a4014420 con 0x7f67bc0f53f0 2023-10-12T22:38:50.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:48.866814+0000 mgr.smithi131.efktma (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:50.696 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.695+0000 7f67c118d700 1 -- 172.21.15.195:0/3378305941 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67bc108110 con 0x7f67bc0f53f0 2023-10-12T22:38:50.696 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.695+0000 7f67b37fe700 1 -- 172.21.15.195:0/3378305941 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f67a402c090 con 0x7f67bc0f53f0 2023-10-12T22:38:50.697 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:50.697 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:50.698 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.697+0000 7f67c118d700 1 -- 172.21.15.195:0/3378305941 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f67a802ff60 msgr2=0x7f67a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:50.698 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.697+0000 7f67c118d700 1 --2- 172.21.15.195:0/3378305941 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f67a802ff60 0x7f67a8032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f67ac0096a0 tx=0x7f67ac006b40).stop 2023-10-12T22:38:50.698 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.697+0000 7f67c118d700 1 -- 172.21.15.195:0/3378305941 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67bc0f53f0 msgr2=0x7f67bc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:50.699 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.697+0000 7f67c118d700 1 --2- 172.21.15.195:0/3378305941 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67bc0f53f0 0x7f67bc1066e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f67a4013510 tx=0x7f67a400b040).stop 2023-10-12T22:38:50.699 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.697+0000 7f67c118d700 1 -- 172.21.15.195:0/3378305941 shutdown_connections 2023-10-12T22:38:50.699 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.697+0000 7f67c118d700 1 --2- 172.21.15.195:0/3378305941 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f67a802ff60 0x7f67a8032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:50.699 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.697+0000 7f67c118d700 1 --2- 172.21.15.195:0/3378305941 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f67bc0f53f0 0x7f67bc1066e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:50.700 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.697+0000 7f67c118d700 1 -- 172.21.15.195:0/3378305941 >> 172.21.15.195:0/3378305941 conn(0x7f67bc0f0b70 msgr2=0x7f67bc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:50.700 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.697+0000 7f67c118d700 1 -- 172.21.15.195:0/3378305941 shutdown_connections 2023-10-12T22:38:50.700 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:50.697+0000 7f67c118d700 1 -- 172.21.15.195:0/3378305941 wait complete. 2023-10-12T22:38:50.701 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:51.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:51 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:50.696667+0000 mon.smithi131 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.195:0/3378305941' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:52.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:52.080 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:52.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:50.867147+0000 mgr.smithi131.efktma (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:53.238 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.237+0000 7f92a359e700 1 -- 172.21.15.195:0/146852030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92a40f51a0 msgr2=0x7f92a40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:53.239 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.237+0000 7f92a359e700 1 --2- 172.21.15.195:0/146852030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92a40f51a0 0x7f92a40f55c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f9294009fb0 tx=0x7f9294009b30).stop 2023-10-12T22:38:53.239 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.237+0000 7f92a359e700 1 -- 172.21.15.195:0/146852030 shutdown_connections 2023-10-12T22:38:53.239 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.237+0000 7f92a359e700 1 --2- 172.21.15.195:0/146852030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92a40f51a0 0x7f92a40f55c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:53.240 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.237+0000 7f92a359e700 1 -- 172.21.15.195:0/146852030 >> 172.21.15.195:0/146852030 conn(0x7f92a40f09a0 msgr2=0x7f92a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:53.240 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.237+0000 7f92a359e700 1 -- 172.21.15.195:0/146852030 shutdown_connections 2023-10-12T22:38:53.240 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.237+0000 7f92a359e700 1 -- 172.21.15.195:0/146852030 wait complete. 2023-10-12T22:38:53.240 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.237+0000 7f92a359e700 1 Processor -- start 2023-10-12T22:38:53.240 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.237+0000 7f92a359e700 1 -- start start 2023-10-12T22:38:53.240 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.238+0000 7f92a359e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92a40f51a0 0x7f92a41086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:53.241 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.238+0000 7f92a359e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a4108bc0 con 0x7f92a40f51a0 2023-10-12T22:38:53.243 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.238+0000 7f92a259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92a40f51a0 0x7f92a41086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:53.243 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.238+0000 7f92a259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92a40f51a0 0x7f92a41086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:53340/0 (socket says 172.21.15.195:53340) 2023-10-12T22:38:53.243 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.238+0000 7f92a259c700 1 -- 172.21.15.195:0/289621620 learned_addr learned my addr 172.21.15.195:0/289621620 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:53.243 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.238+0000 7f92a259c700 1 -- 172.21.15.195:0/289621620 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9294005040 con 0x7f92a40f51a0 2023-10-12T22:38:53.243 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.238+0000 7f92a259c700 1 --2- 172.21.15.195:0/289621620 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92a40f51a0 0x7f92a41086d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f9294009830 tx=0x7f929400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:53.244 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.238+0000 7f9292ffd700 1 -- 172.21.15.195:0/289621620 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9294009320 con 0x7f92a40f51a0 2023-10-12T22:38:53.244 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.238+0000 7f92a359e700 1 -- 172.21.15.195:0/289621620 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a4108dc0 con 0x7f92a40f51a0 2023-10-12T22:38:53.244 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.239+0000 7f9292ffd700 1 -- 172.21.15.195:0/289621620 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92940136c0 con 0x7f92a40f51a0 2023-10-12T22:38:53.244 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.239+0000 7f9292ffd700 1 -- 172.21.15.195:0/289621620 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f929401ee90 con 0x7f92a40f51a0 2023-10-12T22:38:53.244 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.239+0000 7f92a359e700 1 -- 172.21.15.195:0/289621620 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a4109250 con 0x7f92a40f51a0 2023-10-12T22:38:53.245 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.239+0000 7f92a359e700 1 -- 172.21.15.195:0/289621620 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a4045ad0 con 0x7f92a40f51a0 2023-10-12T22:38:53.245 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.239+0000 7f9292ffd700 1 -- 172.21.15.195:0/289621620 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f929401e420 con 0x7f92a40f51a0 2023-10-12T22:38:53.245 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.239+0000 7f9292ffd700 1 --2- 172.21.15.195:0/289621620 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f928c02ff10 0x7f928c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:53.245 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.240+0000 7f9292ffd700 1 -- 172.21.15.195:0/289621620 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9294040fa0 con 0x7f92a40f51a0 2023-10-12T22:38:53.246 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.240+0000 7f92a1d9b700 1 --2- 172.21.15.195:0/289621620 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f928c02ff10 0x7f928c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:53.246 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.240+0000 7f92a1d9b700 1 --2- 172.21.15.195:0/289621620 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f928c02ff10 0x7f928c0323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f92980096a0 tx=0x7f9298006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:53.246 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.244+0000 7f9292ffd700 1 -- 172.21.15.195:0/289621620 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92940095f0 con 0x7f92a40f51a0 2023-10-12T22:38:53.453 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.451+0000 7f92a359e700 1 -- 172.21.15.195:0/289621620 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92a410a0a0 con 0x7f92a40f51a0 2023-10-12T22:38:53.453 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.452+0000 7f9292ffd700 1 -- 172.21.15.195:0/289621620 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f929402c2d0 con 0x7f92a40f51a0 2023-10-12T22:38:53.454 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:53.454 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:53.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.454+0000 7f92a359e700 1 -- 172.21.15.195:0/289621620 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f928c02ff10 msgr2=0x7f928c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:53.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.454+0000 7f92a359e700 1 --2- 172.21.15.195:0/289621620 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f928c02ff10 0x7f928c0323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f92980096a0 tx=0x7f9298006b40).stop 2023-10-12T22:38:53.455 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.454+0000 7f92a359e700 1 -- 172.21.15.195:0/289621620 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92a40f51a0 msgr2=0x7f92a41086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:53.456 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.454+0000 7f92a359e700 1 --2- 172.21.15.195:0/289621620 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92a40f51a0 0x7f92a41086d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f9294009830 tx=0x7f929400b040).stop 2023-10-12T22:38:53.456 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.454+0000 7f92a359e700 1 -- 172.21.15.195:0/289621620 shutdown_connections 2023-10-12T22:38:53.456 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.454+0000 7f92a359e700 1 --2- 172.21.15.195:0/289621620 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f928c02ff10 0x7f928c0323d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:53.456 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.454+0000 7f92a359e700 1 --2- 172.21.15.195:0/289621620 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92a40f51a0 0x7f92a41086d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:53.456 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.454+0000 7f92a359e700 1 -- 172.21.15.195:0/289621620 >> 172.21.15.195:0/289621620 conn(0x7f92a40f09a0 msgr2=0x7f92a405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:53.457 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.454+0000 7f92a359e700 1 -- 172.21.15.195:0/289621620 shutdown_connections 2023-10-12T22:38:53.457 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:53.454+0000 7f92a359e700 1 -- 172.21.15.195:0/289621620 wait complete. 2023-10-12T22:38:53.457 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:54.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:52.867454+0000 mgr.smithi131.efktma (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:54.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:53.453624+0000 mon.smithi131 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.195:0/289621620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:54.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:54.876 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:56.069 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.066+0000 7f2c4db12700 1 -- 172.21.15.195:0/2878937366 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c480f6510 msgr2=0x7f2c480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:56.069 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.066+0000 7f2c4db12700 1 --2- 172.21.15.195:0/2878937366 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c480f6510 0x7f2c480f6930 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f2c30009fb0 tx=0x7f2c30009b30).stop 2023-10-12T22:38:56.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.066+0000 7f2c4db12700 1 -- 172.21.15.195:0/2878937366 shutdown_connections 2023-10-12T22:38:56.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.066+0000 7f2c4db12700 1 --2- 172.21.15.195:0/2878937366 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c480f6510 0x7f2c480f6930 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:56.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.066+0000 7f2c4db12700 1 -- 172.21.15.195:0/2878937366 >> 172.21.15.195:0/2878937366 conn(0x7f2c480f1c70 msgr2=0x7f2c480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:56.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.066+0000 7f2c4db12700 1 -- 172.21.15.195:0/2878937366 shutdown_connections 2023-10-12T22:38:56.070 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.066+0000 7f2c4db12700 1 -- 172.21.15.195:0/2878937366 wait complete. 2023-10-12T22:38:56.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.067+0000 7f2c4db12700 1 Processor -- start 2023-10-12T22:38:56.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.067+0000 7f2c4db12700 1 -- start start 2023-10-12T22:38:56.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.067+0000 7f2c4db12700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c480f6510 0x7f2c4806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:56.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.067+0000 7f2c4db12700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48069e20 con 0x7f2c480f6510 2023-10-12T22:38:56.071 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.067+0000 7f2c477fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c480f6510 0x7f2c4806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:56.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.067+0000 7f2c477fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c480f6510 0x7f2c4806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:53356/0 (socket says 172.21.15.195:53356) 2023-10-12T22:38:56.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.067+0000 7f2c477fe700 1 -- 172.21.15.195:0/2223543579 learned_addr learned my addr 172.21.15.195:0/2223543579 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:56.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.068+0000 7f2c477fe700 1 -- 172.21.15.195:0/2223543579 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c30005040 con 0x7f2c480f6510 2023-10-12T22:38:56.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.068+0000 7f2c477fe700 1 --2- 172.21.15.195:0/2223543579 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c480f6510 0x7f2c4806b9a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f2c30013510 tx=0x7f2c3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:56.072 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.068+0000 7f2c3ffff700 1 -- 172.21.15.195:0/2223543579 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c30009320 con 0x7f2c480f6510 2023-10-12T22:38:56.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.068+0000 7f2c4db12700 1 -- 172.21.15.195:0/2223543579 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c48069fc0 con 0x7f2c480f6510 2023-10-12T22:38:56.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.068+0000 7f2c3ffff700 1 -- 172.21.15.195:0/2223543579 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c30013f00 con 0x7f2c480f6510 2023-10-12T22:38:56.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.068+0000 7f2c3ffff700 1 -- 172.21.15.195:0/2223543579 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c30014d90 con 0x7f2c480f6510 2023-10-12T22:38:56.073 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.068+0000 7f2c4db12700 1 -- 172.21.15.195:0/2223543579 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4806a450 con 0x7f2c480f6510 2023-10-12T22:38:56.074 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.069+0000 7f2c3ffff700 1 -- 172.21.15.195:0/2223543579 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c30014420 con 0x7f2c480f6510 2023-10-12T22:38:56.075 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.069+0000 7f2c4db12700 1 -- 172.21.15.195:0/2223543579 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c48059100 con 0x7f2c480f6510 2023-10-12T22:38:56.075 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.069+0000 7f2c3ffff700 1 --2- 172.21.15.195:0/2223543579 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2c3402ff60 0x7f2c34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:56.075 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.069+0000 7f2c3ffff700 1 -- 172.21.15.195:0/2223543579 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c300410c0 con 0x7f2c480f6510 2023-10-12T22:38:56.075 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.069+0000 7f2c46ffd700 1 --2- 172.21.15.195:0/2223543579 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2c3402ff60 0x7f2c34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:56.076 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.070+0000 7f2c46ffd700 1 --2- 172.21.15.195:0/2223543579 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2c3402ff60 0x7f2c34032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f2c380096a0 tx=0x7f2c38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:56.076 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.074+0000 7f2c3ffff700 1 -- 172.21.15.195:0/2223543579 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c300095f0 con 0x7f2c480f6510 2023-10-12T22:38:56.281 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.279+0000 7f2c4db12700 1 -- 172.21.15.195:0/2223543579 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c480fbf60 con 0x7f2c480f6510 2023-10-12T22:38:56.281 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.280+0000 7f2c3ffff700 1 -- 172.21.15.195:0/2223543579 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2c3001db70 con 0x7f2c480f6510 2023-10-12T22:38:56.282 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:56.282 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:56.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.282+0000 7f2c4db12700 1 -- 172.21.15.195:0/2223543579 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2c3402ff60 msgr2=0x7f2c34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:56.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.282+0000 7f2c4db12700 1 --2- 172.21.15.195:0/2223543579 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2c3402ff60 0x7f2c34032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f2c380096a0 tx=0x7f2c38006b40).stop 2023-10-12T22:38:56.283 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.282+0000 7f2c4db12700 1 -- 172.21.15.195:0/2223543579 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c480f6510 msgr2=0x7f2c4806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:56.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.282+0000 7f2c4db12700 1 --2- 172.21.15.195:0/2223543579 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c480f6510 0x7f2c4806b9a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f2c30013510 tx=0x7f2c3000b040).stop 2023-10-12T22:38:56.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.282+0000 7f2c4db12700 1 -- 172.21.15.195:0/2223543579 shutdown_connections 2023-10-12T22:38:56.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.282+0000 7f2c4db12700 1 --2- 172.21.15.195:0/2223543579 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f2c3402ff60 0x7f2c34032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:56.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.282+0000 7f2c4db12700 1 --2- 172.21.15.195:0/2223543579 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c480f6510 0x7f2c4806b9a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:56.284 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.282+0000 7f2c4db12700 1 -- 172.21.15.195:0/2223543579 >> 172.21.15.195:0/2223543579 conn(0x7f2c480f1c70 msgr2=0x7f2c480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:56.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.282+0000 7f2c4db12700 1 -- 172.21.15.195:0/2223543579 shutdown_connections 2023-10-12T22:38:56.285 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:56.282+0000 7f2c4db12700 1 -- 172.21.15.195:0/2223543579 wait complete. 2023-10-12T22:38:56.285 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:38:56.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:54.867830+0000 mgr.smithi131.efktma (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:57.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:57 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:56.281609+0000 mon.smithi131 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.195:0/2223543579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:38:57.666 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:38:57.667 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:38:58.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:38:58 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:56.868218+0000 mgr.smithi131.efktma (mgr.14162) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:38:58.888 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.884+0000 7f7a9a669700 1 -- 172.21.15.195:0/1442751346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a940f5410 msgr2=0x7f7a940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:58.888 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.884+0000 7f7a9a669700 1 --2- 172.21.15.195:0/1442751346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a940f5410 0x7f7a940f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f7a84009fb0 tx=0x7f7a84009b30).stop 2023-10-12T22:38:58.889 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.884+0000 7f7a9a669700 1 -- 172.21.15.195:0/1442751346 shutdown_connections 2023-10-12T22:38:58.889 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.884+0000 7f7a9a669700 1 --2- 172.21.15.195:0/1442751346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a940f5410 0x7f7a940f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:58.889 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.884+0000 7f7a9a669700 1 -- 172.21.15.195:0/1442751346 >> 172.21.15.195:0/1442751346 conn(0x7f7a940f0b70 msgr2=0x7f7a940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:58.889 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.884+0000 7f7a9a669700 1 -- 172.21.15.195:0/1442751346 shutdown_connections 2023-10-12T22:38:58.890 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.884+0000 7f7a9a669700 1 -- 172.21.15.195:0/1442751346 wait complete. 2023-10-12T22:38:58.890 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.885+0000 7f7a9a669700 1 Processor -- start 2023-10-12T22:38:58.890 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.885+0000 7f7a9a669700 1 -- start start 2023-10-12T22:38:58.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.885+0000 7f7a9a669700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a940f5410 0x7f7a94109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:58.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.885+0000 7f7a9a669700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a94060240 con 0x7f7a940f5410 2023-10-12T22:38:58.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.885+0000 7f7a93fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a940f5410 0x7f7a94109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:58.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.885+0000 7f7a93fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a940f5410 0x7f7a94109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:47152/0 (socket says 172.21.15.195:47152) 2023-10-12T22:38:58.891 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.885+0000 7f7a93fff700 1 -- 172.21.15.195:0/660497324 learned_addr learned my addr 172.21.15.195:0/660497324 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:38:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.886+0000 7f7a93fff700 1 -- 172.21.15.195:0/660497324 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a84005040 con 0x7f7a940f5410 2023-10-12T22:38:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.886+0000 7f7a93fff700 1 --2- 172.21.15.195:0/660497324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a940f5410 0x7f7a94109430 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f7a84013510 tx=0x7f7a8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.886+0000 7f7a90ff9700 1 -- 172.21.15.195:0/660497324 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a84009320 con 0x7f7a940f5410 2023-10-12T22:38:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.886+0000 7f7a9a669700 1 -- 172.21.15.195:0/660497324 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a94060440 con 0x7f7a940f5410 2023-10-12T22:38:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.886+0000 7f7a90ff9700 1 -- 172.21.15.195:0/660497324 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a84013f00 con 0x7f7a940f5410 2023-10-12T22:38:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.886+0000 7f7a90ff9700 1 -- 172.21.15.195:0/660497324 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a84014d90 con 0x7f7a940f5410 2023-10-12T22:38:58.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.886+0000 7f7a9a669700 1 -- 172.21.15.195:0/660497324 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a94109ae0 con 0x7f7a940f5410 2023-10-12T22:38:58.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.887+0000 7f7a90ff9700 1 -- 172.21.15.195:0/660497324 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a840263a0 con 0x7f7a940f5410 2023-10-12T22:38:58.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.887+0000 7f7a90ff9700 1 --2- 172.21.15.195:0/660497324 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7a7c02ffb0 0x7f7a7c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:38:58.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.887+0000 7f7a90ff9700 1 -- 172.21.15.195:0/660497324 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a84040ff0 con 0x7f7a940f5410 2023-10-12T22:38:58.893 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.887+0000 7f7a9a669700 1 -- 172.21.15.195:0/660497324 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a94045ad0 con 0x7f7a940f5410 2023-10-12T22:38:58.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.887+0000 7f7a937fe700 1 --2- 172.21.15.195:0/660497324 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7a7c02ffb0 0x7f7a7c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:38:58.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.888+0000 7f7a937fe700 1 --2- 172.21.15.195:0/660497324 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7a7c02ffb0 0x7f7a7c032470 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f7a880096a0 tx=0x7f7a88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:38:58.894 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:58.892+0000 7f7a90ff9700 1 -- 172.21.15.195:0/660497324 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a84012920 con 0x7f7a940f5410 2023-10-12T22:38:59.096 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:59.096+0000 7f7a9a669700 1 -- 172.21.15.195:0/660497324 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a9410a1f0 con 0x7f7a940f5410 2023-10-12T22:38:59.097 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:59.096+0000 7f7a90ff9700 1 -- 172.21.15.195:0/660497324 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7a8402b020 con 0x7f7a940f5410 2023-10-12T22:38:59.097 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:38:59.097 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:38:59.098 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:59.098+0000 7f7a9a669700 1 -- 172.21.15.195:0/660497324 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7a7c02ffb0 msgr2=0x7f7a7c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:59.098 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:59.098+0000 7f7a9a669700 1 --2- 172.21.15.195:0/660497324 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7a7c02ffb0 0x7f7a7c032470 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f7a880096a0 tx=0x7f7a88006b40).stop 2023-10-12T22:38:59.099 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:59.098+0000 7f7a9a669700 1 -- 172.21.15.195:0/660497324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a940f5410 msgr2=0x7f7a94109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:38:59.099 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:59.098+0000 7f7a9a669700 1 --2- 172.21.15.195:0/660497324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a940f5410 0x7f7a94109430 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f7a84013510 tx=0x7f7a8400b040).stop 2023-10-12T22:38:59.099 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:59.098+0000 7f7a9a669700 1 -- 172.21.15.195:0/660497324 shutdown_connections 2023-10-12T22:38:59.099 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:59.098+0000 7f7a9a669700 1 --2- 172.21.15.195:0/660497324 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f7a7c02ffb0 0x7f7a7c032470 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:59.099 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:59.098+0000 7f7a9a669700 1 --2- 172.21.15.195:0/660497324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a940f5410 0x7f7a94109430 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:38:59.100 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:59.098+0000 7f7a9a669700 1 -- 172.21.15.195:0/660497324 >> 172.21.15.195:0/660497324 conn(0x7f7a940f0b70 msgr2=0x7f7a940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:38:59.100 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:59.098+0000 7f7a9a669700 1 -- 172.21.15.195:0/660497324 shutdown_connections 2023-10-12T22:38:59.100 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:38:59.098+0000 7f7a9a669700 1 -- 172.21.15.195:0/660497324 wait complete. 2023-10-12T22:38:59.100 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:00.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:00.472 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:00.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:38:58.868497+0000 mgr.smithi131.efktma (mgr.14162) 234 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:00.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:38:59.096961+0000 mon.smithi131 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.195:0/660497324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:01.653 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.652+0000 7f9e14a01700 1 -- 172.21.15.195:0/1726533727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e100f51a0 msgr2=0x7f9e100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:01.653 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.652+0000 7f9e14a01700 1 --2- 172.21.15.195:0/1726533727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e100f51a0 0x7f9e100f55c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f9df8009fb0 tx=0x7f9df8009b30).stop 2023-10-12T22:39:01.654 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.652+0000 7f9e14a01700 1 -- 172.21.15.195:0/1726533727 shutdown_connections 2023-10-12T22:39:01.654 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.652+0000 7f9e14a01700 1 --2- 172.21.15.195:0/1726533727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e100f51a0 0x7f9e100f55c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:01.654 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.652+0000 7f9e14a01700 1 -- 172.21.15.195:0/1726533727 >> 172.21.15.195:0/1726533727 conn(0x7f9e100f09a0 msgr2=0x7f9e100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:01.654 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.653+0000 7f9e14a01700 1 -- 172.21.15.195:0/1726533727 shutdown_connections 2023-10-12T22:39:01.654 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.653+0000 7f9e14a01700 1 -- 172.21.15.195:0/1726533727 wait complete. 2023-10-12T22:39:01.656 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.653+0000 7f9e14a01700 1 Processor -- start 2023-10-12T22:39:01.656 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.653+0000 7f9e14a01700 1 -- start start 2023-10-12T22:39:01.656 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.654+0000 7f9e14a01700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e100f51a0 0x7f9e101091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:01.657 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.654+0000 7f9e14a01700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e10060240 con 0x7f9e100f51a0 2023-10-12T22:39:01.657 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.654+0000 7f9e0ed9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e100f51a0 0x7f9e101091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:01.657 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.654+0000 7f9e0ed9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e100f51a0 0x7f9e101091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:47168/0 (socket says 172.21.15.195:47168) 2023-10-12T22:39:01.657 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.654+0000 7f9e0ed9d700 1 -- 172.21.15.195:0/1017890351 learned_addr learned my addr 172.21.15.195:0/1017890351 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:01.657 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.654+0000 7f9e0ed9d700 1 -- 172.21.15.195:0/1017890351 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df8005040 con 0x7f9e100f51a0 2023-10-12T22:39:01.658 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.654+0000 7f9e0ed9d700 1 --2- 172.21.15.195:0/1017890351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e100f51a0 0x7f9e101091b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f9df8013510 tx=0x7f9df800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:01.658 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.655+0000 7f9e077fe700 1 -- 172.21.15.195:0/1017890351 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9df8009320 con 0x7f9e100f51a0 2023-10-12T22:39:01.658 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.655+0000 7f9e077fe700 1 -- 172.21.15.195:0/1017890351 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9df8013f00 con 0x7f9e100f51a0 2023-10-12T22:39:01.658 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.655+0000 7f9e14a01700 1 -- 172.21.15.195:0/1017890351 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e10060440 con 0x7f9e100f51a0 2023-10-12T22:39:01.658 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.655+0000 7f9e077fe700 1 -- 172.21.15.195:0/1017890351 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9df8014dc0 con 0x7f9e100f51a0 2023-10-12T22:39:01.659 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.655+0000 7f9e14a01700 1 -- 172.21.15.195:0/1017890351 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e10109860 con 0x7f9e100f51a0 2023-10-12T22:39:01.659 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.656+0000 7f9e077fe700 1 -- 172.21.15.195:0/1017890351 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9df8014420 con 0x7f9e100f51a0 2023-10-12T22:39:01.659 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.656+0000 7f9e077fe700 1 --2- 172.21.15.195:0/1017890351 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9dfc02ff60 0x7f9dfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:01.659 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.656+0000 7f9e14a01700 1 -- 172.21.15.195:0/1017890351 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e10045ad0 con 0x7f9e100f51a0 2023-10-12T22:39:01.659 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.656+0000 7f9e077fe700 1 -- 172.21.15.195:0/1017890351 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9df8041050 con 0x7f9e100f51a0 2023-10-12T22:39:01.660 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.656+0000 7f9e0e59c700 1 --2- 172.21.15.195:0/1017890351 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9dfc02ff60 0x7f9dfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:01.660 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.657+0000 7f9e0e59c700 1 --2- 172.21.15.195:0/1017890351 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9dfc02ff60 0x7f9dfc032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f9e000096a0 tx=0x7f9e00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:01.661 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.661+0000 7f9e077fe700 1 -- 172.21.15.195:0/1017890351 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9df801cb10 con 0x7f9e100f51a0 2023-10-12T22:39:01.866 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.866+0000 7f9e14a01700 1 -- 172.21.15.195:0/1017890351 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e1010a080 con 0x7f9e100f51a0 2023-10-12T22:39:01.867 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.867+0000 7f9e077fe700 1 -- 172.21.15.195:0/1017890351 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9df801cb10 con 0x7f9e100f51a0 2023-10-12T22:39:01.867 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:01.868 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:01.869 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.868+0000 7f9e14a01700 1 -- 172.21.15.195:0/1017890351 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9dfc02ff60 msgr2=0x7f9dfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:01.869 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.868+0000 7f9e14a01700 1 --2- 172.21.15.195:0/1017890351 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9dfc02ff60 0x7f9dfc032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f9e000096a0 tx=0x7f9e00006b40).stop 2023-10-12T22:39:01.869 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.868+0000 7f9e14a01700 1 -- 172.21.15.195:0/1017890351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e100f51a0 msgr2=0x7f9e101091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:01.869 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.868+0000 7f9e14a01700 1 --2- 172.21.15.195:0/1017890351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e100f51a0 0x7f9e101091b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f9df8013510 tx=0x7f9df800b040).stop 2023-10-12T22:39:01.869 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.868+0000 7f9e14a01700 1 -- 172.21.15.195:0/1017890351 shutdown_connections 2023-10-12T22:39:01.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.868+0000 7f9e14a01700 1 --2- 172.21.15.195:0/1017890351 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9dfc02ff60 0x7f9dfc032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:01.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.868+0000 7f9e14a01700 1 --2- 172.21.15.195:0/1017890351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e100f51a0 0x7f9e101091b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:01.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.868+0000 7f9e14a01700 1 -- 172.21.15.195:0/1017890351 >> 172.21.15.195:0/1017890351 conn(0x7f9e100f09a0 msgr2=0x7f9e100f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:01.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.868+0000 7f9e14a01700 1 -- 172.21.15.195:0/1017890351 shutdown_connections 2023-10-12T22:39:01.870 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:01.868+0000 7f9e14a01700 1 -- 172.21.15.195:0/1017890351 wait complete. 2023-10-12T22:39:01.871 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:02.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:02 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:00.868845+0000 mgr.smithi131.efktma (mgr.14162) 235 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:02.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:02 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:01.867100+0000 mon.smithi131 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.195:0/1017890351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:03.262 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:03.262 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:04.393 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.391+0000 7f9c627c1700 1 -- 172.21.15.195:0/235307910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c5c0601e0 msgr2=0x7f9c5c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:04.393 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.391+0000 7f9c627c1700 1 --2- 172.21.15.195:0/235307910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c5c0601e0 0x7f9c5c0f9190 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f9c44009fb0 tx=0x7f9c44009b30).stop 2023-10-12T22:39:04.393 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.391+0000 7f9c627c1700 1 -- 172.21.15.195:0/235307910 shutdown_connections 2023-10-12T22:39:04.394 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.391+0000 7f9c627c1700 1 --2- 172.21.15.195:0/235307910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c5c0601e0 0x7f9c5c0f9190 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:04.394 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.391+0000 7f9c627c1700 1 -- 172.21.15.195:0/235307910 >> 172.21.15.195:0/235307910 conn(0x7f9c5c0f0b70 msgr2=0x7f9c5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:04.394 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.391+0000 7f9c627c1700 1 -- 172.21.15.195:0/235307910 shutdown_connections 2023-10-12T22:39:04.394 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.391+0000 7f9c627c1700 1 -- 172.21.15.195:0/235307910 wait complete. 2023-10-12T22:39:04.394 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.392+0000 7f9c627c1700 1 Processor -- start 2023-10-12T22:39:04.395 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.392+0000 7f9c627c1700 1 -- start start 2023-10-12T22:39:04.395 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.392+0000 7f9c627c1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c5c0601e0 0x7f9c5c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:04.395 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.392+0000 7f9c627c1700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c106bd0 con 0x7f9c5c0601e0 2023-10-12T22:39:04.395 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.392+0000 7f9c5bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c5c0601e0 0x7f9c5c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:04.395 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.392+0000 7f9c5bfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c5c0601e0 0x7f9c5c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:47188/0 (socket says 172.21.15.195:47188) 2023-10-12T22:39:04.396 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.392+0000 7f9c5bfff700 1 -- 172.21.15.195:0/2899407832 learned_addr learned my addr 172.21.15.195:0/2899407832 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:04.396 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.392+0000 7f9c5bfff700 1 -- 172.21.15.195:0/2899407832 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c44005040 con 0x7f9c5c0601e0 2023-10-12T22:39:04.396 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.393+0000 7f9c5bfff700 1 --2- 172.21.15.195:0/2899407832 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c5c0601e0 0x7f9c5c1066e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f9c44004dd0 tx=0x7f9c44009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:04.396 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.393+0000 7f9c58ff9700 1 -- 172.21.15.195:0/2899407832 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c44018070 con 0x7f9c5c0601e0 2023-10-12T22:39:04.396 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.393+0000 7f9c627c1700 1 -- 172.21.15.195:0/2899407832 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5c106dd0 con 0x7f9c5c0601e0 2023-10-12T22:39:04.397 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.393+0000 7f9c58ff9700 1 -- 172.21.15.195:0/2899407832 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c44013410 con 0x7f9c5c0601e0 2023-10-12T22:39:04.397 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.393+0000 7f9c58ff9700 1 -- 172.21.15.195:0/2899407832 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c4401c3c0 con 0x7f9c5c0601e0 2023-10-12T22:39:04.397 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.393+0000 7f9c627c1700 1 -- 172.21.15.195:0/2899407832 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5c107260 con 0x7f9c5c0601e0 2023-10-12T22:39:04.398 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.394+0000 7f9c58ff9700 1 -- 172.21.15.195:0/2899407832 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c4401d420 con 0x7f9c5c0601e0 2023-10-12T22:39:04.398 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.394+0000 7f9c627c1700 1 -- 172.21.15.195:0/2899407832 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5c045ad0 con 0x7f9c5c0601e0 2023-10-12T22:39:04.398 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.394+0000 7f9c58ff9700 1 --2- 172.21.15.195:0/2899407832 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9c4802ff60 0x7f9c48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:04.398 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.394+0000 7f9c58ff9700 1 -- 172.21.15.195:0/2899407832 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c44041510 con 0x7f9c5c0601e0 2023-10-12T22:39:04.399 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.394+0000 7f9c5b7fe700 1 --2- 172.21.15.195:0/2899407832 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9c4802ff60 0x7f9c48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:04.399 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.394+0000 7f9c5b7fe700 1 --2- 172.21.15.195:0/2899407832 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9c4802ff60 0x7f9c48032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f9c4c0096a0 tx=0x7f9c4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:04.399 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.398+0000 7f9c58ff9700 1 -- 172.21.15.195:0/2899407832 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c4401d820 con 0x7f9c5c0601e0 2023-10-12T22:39:04.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:02.869161+0000 mgr.smithi131.efktma (mgr.14162) 236 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:04.609 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.609+0000 7f9c627c1700 1 -- 172.21.15.195:0/2899407832 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c5c108110 con 0x7f9c5c0601e0 2023-10-12T22:39:04.609 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.609+0000 7f9c58ff9700 1 -- 172.21.15.195:0/2899407832 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9c4401b370 con 0x7f9c5c0601e0 2023-10-12T22:39:04.610 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:04.610 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:04.611 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.611+0000 7f9c627c1700 1 -- 172.21.15.195:0/2899407832 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9c4802ff60 msgr2=0x7f9c48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:04.611 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.611+0000 7f9c627c1700 1 --2- 172.21.15.195:0/2899407832 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9c4802ff60 0x7f9c48032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f9c4c0096a0 tx=0x7f9c4c006b40).stop 2023-10-12T22:39:04.612 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.611+0000 7f9c627c1700 1 -- 172.21.15.195:0/2899407832 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c5c0601e0 msgr2=0x7f9c5c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:04.612 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.611+0000 7f9c627c1700 1 --2- 172.21.15.195:0/2899407832 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c5c0601e0 0x7f9c5c1066e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f9c44004dd0 tx=0x7f9c44009b30).stop 2023-10-12T22:39:04.612 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.611+0000 7f9c627c1700 1 -- 172.21.15.195:0/2899407832 shutdown_connections 2023-10-12T22:39:04.612 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.611+0000 7f9c627c1700 1 --2- 172.21.15.195:0/2899407832 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f9c4802ff60 0x7f9c48032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:04.612 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.611+0000 7f9c627c1700 1 --2- 172.21.15.195:0/2899407832 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c5c0601e0 0x7f9c5c1066e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:04.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.611+0000 7f9c627c1700 1 -- 172.21.15.195:0/2899407832 >> 172.21.15.195:0/2899407832 conn(0x7f9c5c0f0b70 msgr2=0x7f9c5c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:04.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.611+0000 7f9c627c1700 1 -- 172.21.15.195:0/2899407832 shutdown_connections 2023-10-12T22:39:04.613 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:04.611+0000 7f9c627c1700 1 -- 172.21.15.195:0/2899407832 wait complete. 2023-10-12T22:39:04.613 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:05.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:05 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:04.609887+0000 mon.smithi131 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.195:0/2899407832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:05.997 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:05.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:06.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:04.869490+0000 mgr.smithi131.efktma (mgr.14162) 237 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:07.136 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.135+0000 7f9528fd3700 1 -- 172.21.15.195:0/3458900919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95240f5410 msgr2=0x7f95240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:07.137 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.135+0000 7f9528fd3700 1 --2- 172.21.15.195:0/3458900919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95240f5410 0x7f95240f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f950c009fb0 tx=0x7f950c009b30).stop 2023-10-12T22:39:07.137 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.135+0000 7f9528fd3700 1 -- 172.21.15.195:0/3458900919 shutdown_connections 2023-10-12T22:39:07.137 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.135+0000 7f9528fd3700 1 --2- 172.21.15.195:0/3458900919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95240f5410 0x7f95240f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:07.137 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.135+0000 7f9528fd3700 1 -- 172.21.15.195:0/3458900919 >> 172.21.15.195:0/3458900919 conn(0x7f95240f0b70 msgr2=0x7f95240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:07.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.135+0000 7f9528fd3700 1 -- 172.21.15.195:0/3458900919 shutdown_connections 2023-10-12T22:39:07.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.135+0000 7f9528fd3700 1 -- 172.21.15.195:0/3458900919 wait complete. 2023-10-12T22:39:07.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.136+0000 7f9528fd3700 1 Processor -- start 2023-10-12T22:39:07.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.136+0000 7f9528fd3700 1 -- start start 2023-10-12T22:39:07.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.136+0000 7f9528fd3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95240f5410 0x7f9524106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:07.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.136+0000 7f9528fd3700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9524106c10 con 0x7f95240f5410 2023-10-12T22:39:07.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.136+0000 7f952259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95240f5410 0x7f9524106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:07.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.136+0000 7f952259c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95240f5410 0x7f9524106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:47212/0 (socket says 172.21.15.195:47212) 2023-10-12T22:39:07.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.136+0000 7f952259c700 1 -- 172.21.15.195:0/2076561612 learned_addr learned my addr 172.21.15.195:0/2076561612 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:07.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.137+0000 7f952259c700 1 -- 172.21.15.195:0/2076561612 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c005040 con 0x7f95240f5410 2023-10-12T22:39:07.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.137+0000 7f952259c700 1 --2- 172.21.15.195:0/2076561612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95240f5410 0x7f9524106720 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f950c014000 tx=0x7f950c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:07.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.137+0000 7f951affd700 1 -- 172.21.15.195:0/2076561612 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f950c0095f0 con 0x7f95240f5410 2023-10-12T22:39:07.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.137+0000 7f9528fd3700 1 -- 172.21.15.195:0/2076561612 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9524106e10 con 0x7f95240f5410 2023-10-12T22:39:07.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.137+0000 7f951affd700 1 -- 172.21.15.195:0/2076561612 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f950c007e70 con 0x7f95240f5410 2023-10-12T22:39:07.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.137+0000 7f951affd700 1 -- 172.21.15.195:0/2076561612 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f950c014ea0 con 0x7f95240f5410 2023-10-12T22:39:07.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.137+0000 7f9528fd3700 1 -- 172.21.15.195:0/2076561612 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95241072a0 con 0x7f95240f5410 2023-10-12T22:39:07.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.138+0000 7f951affd700 1 -- 172.21.15.195:0/2076561612 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f950c01d4e0 con 0x7f95240f5410 2023-10-12T22:39:07.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.138+0000 7f951affd700 1 --2- 172.21.15.195:0/2076561612 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f951002ff60 0x7f9510032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:07.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.138+0000 7f951affd700 1 -- 172.21.15.195:0/2076561612 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f950c0408c0 con 0x7f95240f5410 2023-10-12T22:39:07.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.138+0000 7f9528fd3700 1 -- 172.21.15.195:0/2076561612 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9524045ad0 con 0x7f95240f5410 2023-10-12T22:39:07.143 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.138+0000 7f9521d9b700 1 --2- 172.21.15.195:0/2076561612 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f951002ff60 0x7f9510032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:07.143 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.138+0000 7f9521d9b700 1 --2- 172.21.15.195:0/2076561612 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f951002ff60 0x7f9510032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f95140096a0 tx=0x7f9514006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:07.143 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.143+0000 7f951affd700 1 -- 172.21.15.195:0/2076561612 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f950c00f8f0 con 0x7f95240f5410 2023-10-12T22:39:07.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.349+0000 7f9528fd3700 1 -- 172.21.15.195:0/2076561612 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95240fae60 con 0x7f95240f5410 2023-10-12T22:39:07.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.350+0000 7f951affd700 1 -- 172.21.15.195:0/2076561612 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f95240fae60 con 0x7f95240f5410 2023-10-12T22:39:07.351 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:07.351 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:07.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.351+0000 7f9528fd3700 1 -- 172.21.15.195:0/2076561612 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f951002ff60 msgr2=0x7f9510032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:07.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.352+0000 7f9528fd3700 1 --2- 172.21.15.195:0/2076561612 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f951002ff60 0x7f9510032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f95140096a0 tx=0x7f9514006b40).stop 2023-10-12T22:39:07.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.352+0000 7f9528fd3700 1 -- 172.21.15.195:0/2076561612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95240f5410 msgr2=0x7f9524106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:07.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.352+0000 7f9528fd3700 1 --2- 172.21.15.195:0/2076561612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95240f5410 0x7f9524106720 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f950c014000 tx=0x7f950c00b040).stop 2023-10-12T22:39:07.352 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.352+0000 7f9528fd3700 1 -- 172.21.15.195:0/2076561612 shutdown_connections 2023-10-12T22:39:07.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.352+0000 7f9528fd3700 1 --2- 172.21.15.195:0/2076561612 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f951002ff60 0x7f9510032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:07.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.352+0000 7f9528fd3700 1 --2- 172.21.15.195:0/2076561612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95240f5410 0x7f9524106720 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:07.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.352+0000 7f9528fd3700 1 -- 172.21.15.195:0/2076561612 >> 172.21.15.195:0/2076561612 conn(0x7f95240f0b70 msgr2=0x7f952405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:07.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.352+0000 7f9528fd3700 1 -- 172.21.15.195:0/2076561612 shutdown_connections 2023-10-12T22:39:07.353 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:07.352+0000 7f9528fd3700 1 -- 172.21.15.195:0/2076561612 wait complete. 2023-10-12T22:39:07.354 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:08.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:06.869758+0000 mgr.smithi131.efktma (mgr.14162) 238 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:08.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:07.350504+0000 mon.smithi131 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.195:0/2076561612' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:08.776 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:08.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:09.963 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.961+0000 7ff1536b0700 1 -- 172.21.15.195:0/1551860583 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff14c0f5410 msgr2=0x7ff14c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:09.963 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.961+0000 7ff1536b0700 1 --2- 172.21.15.195:0/1551860583 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff14c0f5410 0x7ff14c0f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7ff13c009fb0 tx=0x7ff13c009b30).stop 2023-10-12T22:39:09.963 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.961+0000 7ff1536b0700 1 -- 172.21.15.195:0/1551860583 shutdown_connections 2023-10-12T22:39:09.963 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.961+0000 7ff1536b0700 1 --2- 172.21.15.195:0/1551860583 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff14c0f5410 0x7ff14c0f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:09.964 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.961+0000 7ff1536b0700 1 -- 172.21.15.195:0/1551860583 >> 172.21.15.195:0/1551860583 conn(0x7ff14c0f0b70 msgr2=0x7ff14c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:09.964 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.962+0000 7ff1536b0700 1 -- 172.21.15.195:0/1551860583 shutdown_connections 2023-10-12T22:39:09.964 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.962+0000 7ff1536b0700 1 -- 172.21.15.195:0/1551860583 wait complete. 2023-10-12T22:39:09.964 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.962+0000 7ff1536b0700 1 Processor -- start 2023-10-12T22:39:09.964 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.962+0000 7ff1536b0700 1 -- start start 2023-10-12T22:39:09.965 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.962+0000 7ff1536b0700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff14c0f5410 0x7ff14c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:09.965 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.962+0000 7ff1536b0700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c060280 con 0x7ff14c0f5410 2023-10-12T22:39:09.965 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.962+0000 7ff15144c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff14c0f5410 0x7ff14c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:09.965 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.962+0000 7ff15144c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff14c0f5410 0x7ff14c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:37642/0 (socket says 172.21.15.195:37642) 2023-10-12T22:39:09.965 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.962+0000 7ff15144c700 1 -- 172.21.15.195:0/517570612 learned_addr learned my addr 172.21.15.195:0/517570612 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:09.966 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.963+0000 7ff15144c700 1 -- 172.21.15.195:0/517570612 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c005040 con 0x7ff14c0f5410 2023-10-12T22:39:09.966 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.963+0000 7ff15144c700 1 --2- 172.21.15.195:0/517570612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff14c0f5410 0x7ff14c1093f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7ff13c013510 tx=0x7ff13c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:09.967 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.963+0000 7ff141ffb700 1 -- 172.21.15.195:0/517570612 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff13c009320 con 0x7ff14c0f5410 2023-10-12T22:39:09.967 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.963+0000 7ff1536b0700 1 -- 172.21.15.195:0/517570612 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c060480 con 0x7ff14c0f5410 2023-10-12T22:39:09.968 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.963+0000 7ff141ffb700 1 -- 172.21.15.195:0/517570612 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff13c013f00 con 0x7ff14c0f5410 2023-10-12T22:39:09.968 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.963+0000 7ff141ffb700 1 -- 172.21.15.195:0/517570612 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff13c014d90 con 0x7ff14c0f5410 2023-10-12T22:39:09.968 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.963+0000 7ff1536b0700 1 -- 172.21.15.195:0/517570612 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c109b20 con 0x7ff14c0f5410 2023-10-12T22:39:09.969 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.964+0000 7ff141ffb700 1 -- 172.21.15.195:0/517570612 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff13c0095f0 con 0x7ff14c0f5410 2023-10-12T22:39:09.969 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.964+0000 7ff1536b0700 1 -- 172.21.15.195:0/517570612 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c045ad0 con 0x7ff14c0f5410 2023-10-12T22:39:09.969 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.964+0000 7ff141ffb700 1 --2- 172.21.15.195:0/517570612 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff13802ff60 0x7ff138032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:09.969 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.964+0000 7ff141ffb700 1 -- 172.21.15.195:0/517570612 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff13c0410c0 con 0x7ff14c0f5410 2023-10-12T22:39:09.969 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.965+0000 7ff150c4b700 1 --2- 172.21.15.195:0/517570612 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff13802ff60 0x7ff138032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:09.970 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.965+0000 7ff150c4b700 1 --2- 172.21.15.195:0/517570612 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff13802ff60 0x7ff138032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7ff1480096a0 tx=0x7ff148006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:09.970 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:09.969+0000 7ff141ffb700 1 -- 172.21.15.195:0/517570612 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff13c01d7f0 con 0x7ff14c0f5410 2023-10-12T22:39:10.176 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:10.175+0000 7ff1536b0700 1 -- 172.21.15.195:0/517570612 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff14c10a340 con 0x7ff14c0f5410 2023-10-12T22:39:10.176 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:10.176+0000 7ff141ffb700 1 -- 172.21.15.195:0/517570612 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff13c010230 con 0x7ff14c0f5410 2023-10-12T22:39:10.177 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:10.177 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:10.178 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:10.177+0000 7ff1536b0700 1 -- 172.21.15.195:0/517570612 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff13802ff60 msgr2=0x7ff138032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:10.178 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:10.177+0000 7ff1536b0700 1 --2- 172.21.15.195:0/517570612 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff13802ff60 0x7ff138032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7ff1480096a0 tx=0x7ff148006b40).stop 2023-10-12T22:39:10.178 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:10.178+0000 7ff1536b0700 1 -- 172.21.15.195:0/517570612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff14c0f5410 msgr2=0x7ff14c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:10.179 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:10.178+0000 7ff1536b0700 1 --2- 172.21.15.195:0/517570612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff14c0f5410 0x7ff14c1093f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7ff13c013510 tx=0x7ff13c00b040).stop 2023-10-12T22:39:10.179 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:10.178+0000 7ff1536b0700 1 -- 172.21.15.195:0/517570612 shutdown_connections 2023-10-12T22:39:10.179 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:10.178+0000 7ff1536b0700 1 --2- 172.21.15.195:0/517570612 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ff13802ff60 0x7ff138032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:10.179 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:10.178+0000 7ff1536b0700 1 --2- 172.21.15.195:0/517570612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff14c0f5410 0x7ff14c1093f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:10.179 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:10.178+0000 7ff1536b0700 1 -- 172.21.15.195:0/517570612 >> 172.21.15.195:0/517570612 conn(0x7ff14c0f0b70 msgr2=0x7ff14c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:10.180 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:10.178+0000 7ff1536b0700 1 -- 172.21.15.195:0/517570612 shutdown_connections 2023-10-12T22:39:10.180 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:10.178+0000 7ff1536b0700 1 -- 172.21.15.195:0/517570612 wait complete. 2023-10-12T22:39:10.180 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:10.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:08.870014+0000 mgr.smithi131.efktma (mgr.14162) 239 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:11.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:10.176396+0000 mon.smithi131 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.195:0/517570612' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:11.557 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:11.558 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:12.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:10.870320+0000 mgr.smithi131.efktma (mgr.14162) 240 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:12.762 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.760+0000 7f86b8dc9700 1 -- 172.21.15.195:0/3253525640 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f86b40f51a0 msgr2=0x7f86b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:12.762 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.760+0000 7f86b8dc9700 1 --2- 172.21.15.195:0/3253525640 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f86b40f51a0 0x7f86b40f55c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f86a4009fb0 tx=0x7f86a4009b30).stop 2023-10-12T22:39:12.762 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.760+0000 7f86b8dc9700 1 -- 172.21.15.195:0/3253525640 shutdown_connections 2023-10-12T22:39:12.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.760+0000 7f86b8dc9700 1 --2- 172.21.15.195:0/3253525640 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f86b40f51a0 0x7f86b40f55c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:12.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.760+0000 7f86b8dc9700 1 -- 172.21.15.195:0/3253525640 >> 172.21.15.195:0/3253525640 conn(0x7f86b40f09a0 msgr2=0x7f86b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:12.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.760+0000 7f86b8dc9700 1 -- 172.21.15.195:0/3253525640 shutdown_connections 2023-10-12T22:39:12.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.760+0000 7f86b8dc9700 1 -- 172.21.15.195:0/3253525640 wait complete. 2023-10-12T22:39:12.763 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.760+0000 7f86b8dc9700 1 Processor -- start 2023-10-12T22:39:12.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.760+0000 7f86b8dc9700 1 -- start start 2023-10-12T22:39:12.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.760+0000 7f86b8dc9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f86b40f51a0 0x7f86b41064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:12.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.760+0000 7f86b8dc9700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b41069d0 con 0x7f86b40f51a0 2023-10-12T22:39:12.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.761+0000 7f86b37fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f86b40f51a0 0x7f86b41064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:12.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.761+0000 7f86b37fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f86b40f51a0 0x7f86b41064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:37666/0 (socket says 172.21.15.195:37666) 2023-10-12T22:39:12.764 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.761+0000 7f86b37fe700 1 -- 172.21.15.195:0/2561442119 learned_addr learned my addr 172.21.15.195:0/2561442119 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:12.766 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.761+0000 7f86b37fe700 1 -- 172.21.15.195:0/2561442119 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a4005040 con 0x7f86b40f51a0 2023-10-12T22:39:12.766 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.761+0000 7f86b37fe700 1 --2- 172.21.15.195:0/2561442119 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f86b40f51a0 0x7f86b41064e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f86a4013510 tx=0x7f86a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:12.766 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.761+0000 7f869bfff700 1 -- 172.21.15.195:0/2561442119 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86a4009320 con 0x7f86b40f51a0 2023-10-12T22:39:12.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.761+0000 7f86b8dc9700 1 -- 172.21.15.195:0/2561442119 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b4106bd0 con 0x7f86b40f51a0 2023-10-12T22:39:12.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.761+0000 7f869bfff700 1 -- 172.21.15.195:0/2561442119 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86a4013f00 con 0x7f86b40f51a0 2023-10-12T22:39:12.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.761+0000 7f869bfff700 1 -- 172.21.15.195:0/2561442119 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86a4014dc0 con 0x7f86b40f51a0 2023-10-12T22:39:12.767 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.761+0000 7f86b8dc9700 1 -- 172.21.15.195:0/2561442119 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b4107060 con 0x7f86b40f51a0 2023-10-12T22:39:12.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.762+0000 7f869bfff700 1 -- 172.21.15.195:0/2561442119 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86a40263a0 con 0x7f86b40f51a0 2023-10-12T22:39:12.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.762+0000 7f86b8dc9700 1 -- 172.21.15.195:0/2561442119 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b4045ad0 con 0x7f86b40f51a0 2023-10-12T22:39:12.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.763+0000 7f869bfff700 1 --2- 172.21.15.195:0/2561442119 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f869c02ffb0 0x7f869c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:12.768 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.763+0000 7f869bfff700 1 -- 172.21.15.195:0/2561442119 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86a4041090 con 0x7f86b40f51a0 2023-10-12T22:39:12.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.763+0000 7f86b2ffd700 1 --2- 172.21.15.195:0/2561442119 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f869c02ffb0 0x7f869c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:12.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.763+0000 7f86b2ffd700 1 --2- 172.21.15.195:0/2561442119 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f869c02ffb0 0x7f869c032470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f86a80096a0 tx=0x7f86a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:12.769 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.767+0000 7f869bfff700 1 -- 172.21.15.195:0/2561442119 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86a406d2b0 con 0x7f86b40f51a0 2023-10-12T22:39:12.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.974+0000 7f86b8dc9700 1 -- 172.21.15.195:0/2561442119 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86b4107f10 con 0x7f86b40f51a0 2023-10-12T22:39:12.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.974+0000 7f869bfff700 1 -- 172.21.15.195:0/2561442119 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f86a40126f0 con 0x7f86b40f51a0 2023-10-12T22:39:12.976 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:12.976 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:12.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.976+0000 7f86b8dc9700 1 -- 172.21.15.195:0/2561442119 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f869c02ffb0 msgr2=0x7f869c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:12.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.976+0000 7f86b8dc9700 1 --2- 172.21.15.195:0/2561442119 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f869c02ffb0 0x7f869c032470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f86a80096a0 tx=0x7f86a8006b40).stop 2023-10-12T22:39:12.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.976+0000 7f86b8dc9700 1 -- 172.21.15.195:0/2561442119 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f86b40f51a0 msgr2=0x7f86b41064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:12.978 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.976+0000 7f86b8dc9700 1 --2- 172.21.15.195:0/2561442119 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f86b40f51a0 0x7f86b41064e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f86a4013510 tx=0x7f86a400b040).stop 2023-10-12T22:39:12.978 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.976+0000 7f86b8dc9700 1 -- 172.21.15.195:0/2561442119 shutdown_connections 2023-10-12T22:39:12.978 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.976+0000 7f86b8dc9700 1 --2- 172.21.15.195:0/2561442119 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f869c02ffb0 0x7f869c032470 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:12.978 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.976+0000 7f86b8dc9700 1 --2- 172.21.15.195:0/2561442119 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f86b40f51a0 0x7f86b41064e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:12.979 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.976+0000 7f86b8dc9700 1 -- 172.21.15.195:0/2561442119 >> 172.21.15.195:0/2561442119 conn(0x7f86b40f09a0 msgr2=0x7f86b405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:12.979 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.976+0000 7f86b8dc9700 1 -- 172.21.15.195:0/2561442119 shutdown_connections 2023-10-12T22:39:12.979 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:12.976+0000 7f86b8dc9700 1 -- 172.21.15.195:0/2561442119 wait complete. 2023-10-12T22:39:12.979 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:13.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:13 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:12.975899+0000 mon.smithi131 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.195:0/2561442119' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:14.356 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:14.356 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:14.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:14 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:12.870636+0000 mgr.smithi131.efktma (mgr.14162) 241 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:15.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.504+0000 7f1f20f99700 1 -- 172.21.15.195:0/274824107 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f1c0f5410 msgr2=0x7f1f1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:15.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.504+0000 7f1f20f99700 1 --2- 172.21.15.195:0/274824107 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f1c0f5410 0x7f1f1c0f5830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f1f0c009fb0 tx=0x7f1f0c009b30).stop 2023-10-12T22:39:15.507 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.504+0000 7f1f20f99700 1 -- 172.21.15.195:0/274824107 shutdown_connections 2023-10-12T22:39:15.508 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.504+0000 7f1f20f99700 1 --2- 172.21.15.195:0/274824107 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f1c0f5410 0x7f1f1c0f5830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:15.508 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.504+0000 7f1f20f99700 1 -- 172.21.15.195:0/274824107 >> 172.21.15.195:0/274824107 conn(0x7f1f1c0f0b70 msgr2=0x7f1f1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:15.508 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.504+0000 7f1f20f99700 1 -- 172.21.15.195:0/274824107 shutdown_connections 2023-10-12T22:39:15.508 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.504+0000 7f1f20f99700 1 -- 172.21.15.195:0/274824107 wait complete. 2023-10-12T22:39:15.508 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.505+0000 7f1f20f99700 1 Processor -- start 2023-10-12T22:39:15.509 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.505+0000 7f1f20f99700 1 -- start start 2023-10-12T22:39:15.509 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.505+0000 7f1f20f99700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f1c0f5410 0x7f1f1c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:15.509 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.505+0000 7f1f20f99700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c106c20 con 0x7f1f1c0f5410 2023-10-12T22:39:15.509 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.505+0000 7f1f1a59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f1c0f5410 0x7f1f1c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:15.509 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.505+0000 7f1f1a59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f1c0f5410 0x7f1f1c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:37680/0 (socket says 172.21.15.195:37680) 2023-10-12T22:39:15.510 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.505+0000 7f1f1a59c700 1 -- 172.21.15.195:0/3018259595 learned_addr learned my addr 172.21.15.195:0/3018259595 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:15.510 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.506+0000 7f1f1a59c700 1 -- 172.21.15.195:0/3018259595 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0c005040 con 0x7f1f1c0f5410 2023-10-12T22:39:15.510 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.506+0000 7f1f1a59c700 1 --2- 172.21.15.195:0/3018259595 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f1c0f5410 0x7f1f1c106730 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f1f0c013510 tx=0x7f1f0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:15.510 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.506+0000 7f1f0affd700 1 -- 172.21.15.195:0/3018259595 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f0c009320 con 0x7f1f1c0f5410 2023-10-12T22:39:15.510 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.506+0000 7f1f20f99700 1 -- 172.21.15.195:0/3018259595 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c106e20 con 0x7f1f1c0f5410 2023-10-12T22:39:15.511 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.506+0000 7f1f0affd700 1 -- 172.21.15.195:0/3018259595 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f0c013f00 con 0x7f1f1c0f5410 2023-10-12T22:39:15.511 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.506+0000 7f1f0affd700 1 -- 172.21.15.195:0/3018259595 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f0c014d90 con 0x7f1f1c0f5410 2023-10-12T22:39:15.511 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.506+0000 7f1f20f99700 1 -- 172.21.15.195:0/3018259595 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c1072b0 con 0x7f1f1c0f5410 2023-10-12T22:39:15.512 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.507+0000 7f1f0affd700 1 -- 172.21.15.195:0/3018259595 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f0c0263a0 con 0x7f1f1c0f5410 2023-10-12T22:39:15.512 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.507+0000 7f1f0affd700 1 --2- 172.21.15.195:0/3018259595 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1f0402ff60 0x7f1f04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:15.513 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.507+0000 7f1f0affd700 1 -- 172.21.15.195:0/3018259595 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f0c0413a0 con 0x7f1f1c0f5410 2023-10-12T22:39:15.513 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.507+0000 7f1f20f99700 1 -- 172.21.15.195:0/3018259595 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c045ad0 con 0x7f1f1c0f5410 2023-10-12T22:39:15.513 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.507+0000 7f1f19d9b700 1 --2- 172.21.15.195:0/3018259595 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1f0402ff60 0x7f1f04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:15.513 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.507+0000 7f1f19d9b700 1 --2- 172.21.15.195:0/3018259595 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1f0402ff60 0x7f1f04032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f1f100096a0 tx=0x7f1f10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:15.514 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.512+0000 7f1f0affd700 1 -- 172.21.15.195:0/3018259595 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f0c012920 con 0x7f1f1c0f5410 2023-10-12T22:39:15.718 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.717+0000 7f1f20f99700 1 -- 172.21.15.195:0/3018259595 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f1c0fae60 con 0x7f1f1c0f5410 2023-10-12T22:39:15.719 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.718+0000 7f1f0affd700 1 -- 172.21.15.195:0/3018259595 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1f0c02b030 con 0x7f1f1c0f5410 2023-10-12T22:39:15.719 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:15.719 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:15.720 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.719+0000 7f1f20f99700 1 -- 172.21.15.195:0/3018259595 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1f0402ff60 msgr2=0x7f1f04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:15.721 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.719+0000 7f1f20f99700 1 --2- 172.21.15.195:0/3018259595 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1f0402ff60 0x7f1f04032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f1f100096a0 tx=0x7f1f10006b40).stop 2023-10-12T22:39:15.721 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.719+0000 7f1f20f99700 1 -- 172.21.15.195:0/3018259595 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f1c0f5410 msgr2=0x7f1f1c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:15.721 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.719+0000 7f1f20f99700 1 --2- 172.21.15.195:0/3018259595 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f1c0f5410 0x7f1f1c106730 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f1f0c013510 tx=0x7f1f0c00b040).stop 2023-10-12T22:39:15.721 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.719+0000 7f1f20f99700 1 -- 172.21.15.195:0/3018259595 shutdown_connections 2023-10-12T22:39:15.721 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.719+0000 7f1f20f99700 1 --2- 172.21.15.195:0/3018259595 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f1f0402ff60 0x7f1f04032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:15.722 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.719+0000 7f1f20f99700 1 --2- 172.21.15.195:0/3018259595 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f1c0f5410 0x7f1f1c106730 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:15.722 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.719+0000 7f1f20f99700 1 -- 172.21.15.195:0/3018259595 >> 172.21.15.195:0/3018259595 conn(0x7f1f1c0f0b70 msgr2=0x7f1f1c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:15.722 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.719+0000 7f1f20f99700 1 -- 172.21.15.195:0/3018259595 shutdown_connections 2023-10-12T22:39:15.722 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:15.719+0000 7f1f20f99700 1 -- 172.21.15.195:0/3018259595 wait complete. 2023-10-12T22:39:15.723 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:16.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:16 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:14.870997+0000 mgr.smithi131.efktma (mgr.14162) 242 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:16.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:16 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:15.718972+0000 mon.smithi131 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.195:0/3018259595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:17.098 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:17.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:18.210 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.208+0000 7fa24b45d700 1 -- 172.21.15.195:0/2432787746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa2440f62a0 msgr2=0x7fa2440f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:18.210 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.208+0000 7fa24b45d700 1 --2- 172.21.15.195:0/2432787746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa2440f62a0 0x7fa2440f66c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fa240009fb0 tx=0x7fa240009b30).stop 2023-10-12T22:39:18.210 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.208+0000 7fa24b45d700 1 -- 172.21.15.195:0/2432787746 shutdown_connections 2023-10-12T22:39:18.210 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.208+0000 7fa24b45d700 1 --2- 172.21.15.195:0/2432787746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa2440f62a0 0x7fa2440f66c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:18.211 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.208+0000 7fa24b45d700 1 -- 172.21.15.195:0/2432787746 >> 172.21.15.195:0/2432787746 conn(0x7fa2440f1ac0 msgr2=0x7fa2440f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:18.211 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.208+0000 7fa24b45d700 1 -- 172.21.15.195:0/2432787746 shutdown_connections 2023-10-12T22:39:18.211 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.208+0000 7fa24b45d700 1 -- 172.21.15.195:0/2432787746 wait complete. 2023-10-12T22:39:18.213 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.209+0000 7fa24b45d700 1 Processor -- start 2023-10-12T22:39:18.213 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.209+0000 7fa24b45d700 1 -- start start 2023-10-12T22:39:18.213 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.209+0000 7fa24b45d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa2440f62a0 0x7fa2441064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:18.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.209+0000 7fa24b45d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa244106990 con 0x7fa2440f62a0 2023-10-12T22:39:18.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.209+0000 7fa24a45b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa2440f62a0 0x7fa2441064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:18.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.209+0000 7fa24a45b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa2440f62a0 0x7fa2441064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:37704/0 (socket says 172.21.15.195:37704) 2023-10-12T22:39:18.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.209+0000 7fa24a45b700 1 -- 172.21.15.195:0/675123393 learned_addr learned my addr 172.21.15.195:0/675123393 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:18.214 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.210+0000 7fa24a45b700 1 -- 172.21.15.195:0/675123393 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa240005040 con 0x7fa2440f62a0 2023-10-12T22:39:18.215 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.210+0000 7fa24a45b700 1 --2- 172.21.15.195:0/675123393 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa2440f62a0 0x7fa2441064a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fa240013510 tx=0x7fa24000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:18.215 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.210+0000 7fa236ffd700 1 -- 172.21.15.195:0/675123393 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa240009320 con 0x7fa2440f62a0 2023-10-12T22:39:18.215 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.210+0000 7fa236ffd700 1 -- 172.21.15.195:0/675123393 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa240013f00 con 0x7fa2440f62a0 2023-10-12T22:39:18.215 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.210+0000 7fa24b45d700 1 -- 172.21.15.195:0/675123393 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa244106b90 con 0x7fa2440f62a0 2023-10-12T22:39:18.215 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.210+0000 7fa236ffd700 1 -- 172.21.15.195:0/675123393 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa240014dc0 con 0x7fa2440f62a0 2023-10-12T22:39:18.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.210+0000 7fa24b45d700 1 -- 172.21.15.195:0/675123393 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa244107020 con 0x7fa2440f62a0 2023-10-12T22:39:18.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.211+0000 7fa236ffd700 1 -- 172.21.15.195:0/675123393 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2400095f0 con 0x7fa2440f62a0 2023-10-12T22:39:18.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.211+0000 7fa24b45d700 1 -- 172.21.15.195:0/675123393 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa244045ad0 con 0x7fa2440f62a0 2023-10-12T22:39:18.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.211+0000 7fa236ffd700 1 --2- 172.21.15.195:0/675123393 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa23002ff60 0x7fa230032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:18.216 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.211+0000 7fa236ffd700 1 -- 172.21.15.195:0/675123393 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2400410c0 con 0x7fa2440f62a0 2023-10-12T22:39:18.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.211+0000 7fa249c5a700 1 --2- 172.21.15.195:0/675123393 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa23002ff60 0x7fa230032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:18.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.212+0000 7fa249c5a700 1 --2- 172.21.15.195:0/675123393 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa23002ff60 0x7fa230032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fa2380096a0 tx=0x7fa238006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:18.217 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.215+0000 7fa236ffd700 1 -- 172.21.15.195:0/675123393 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa240014420 con 0x7fa2440f62a0 2023-10-12T22:39:18.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.422+0000 7fa24b45d700 1 -- 172.21.15.195:0/675123393 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa244107ed0 con 0x7fa2440f62a0 2023-10-12T22:39:18.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.423+0000 7fa236ffd700 1 -- 172.21.15.195:0/675123393 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa240010230 con 0x7fa2440f62a0 2023-10-12T22:39:18.432 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:18.432 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:18.433 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.424+0000 7fa24b45d700 1 -- 172.21.15.195:0/675123393 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa23002ff60 msgr2=0x7fa230032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:18.434 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.424+0000 7fa24b45d700 1 --2- 172.21.15.195:0/675123393 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa23002ff60 0x7fa230032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fa2380096a0 tx=0x7fa238006b40).stop 2023-10-12T22:39:18.434 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.424+0000 7fa24b45d700 1 -- 172.21.15.195:0/675123393 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa2440f62a0 msgr2=0x7fa2441064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:18.435 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.424+0000 7fa24b45d700 1 --2- 172.21.15.195:0/675123393 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa2440f62a0 0x7fa2441064a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fa240013510 tx=0x7fa24000b040).stop 2023-10-12T22:39:18.435 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.424+0000 7fa24b45d700 1 -- 172.21.15.195:0/675123393 shutdown_connections 2023-10-12T22:39:18.435 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.424+0000 7fa24b45d700 1 --2- 172.21.15.195:0/675123393 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa23002ff60 0x7fa230032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:18.436 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.424+0000 7fa24b45d700 1 --2- 172.21.15.195:0/675123393 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa2440f62a0 0x7fa2441064a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:18.436 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.424+0000 7fa24b45d700 1 -- 172.21.15.195:0/675123393 >> 172.21.15.195:0/675123393 conn(0x7fa2440f1ac0 msgr2=0x7fa2440f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:18.437 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.424+0000 7fa24b45d700 1 -- 172.21.15.195:0/675123393 shutdown_connections 2023-10-12T22:39:18.437 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:18.425+0000 7fa24b45d700 1 -- 172.21.15.195:0/675123393 wait complete. 2023-10-12T22:39:18.438 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:18.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:18 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:16.871351+0000 mgr.smithi131.efktma (mgr.14162) 243 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:19.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:19 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:18.423974+0000 mon.smithi131 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.195:0/675123393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:19.803 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:19.804 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:20.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:20 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:18.871732+0000 mgr.smithi131.efktma (mgr.14162) 244 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:21.008 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.007+0000 7f8d86a30700 1 -- 172.21.15.195:0/3832697292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d80060220 msgr2=0x7f8d800fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:21.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.007+0000 7f8d86a30700 1 --2- 172.21.15.195:0/3832697292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d80060220 0x7f8d800fb550 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f8d70009fb0 tx=0x7f8d70009b30).stop 2023-10-12T22:39:21.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.007+0000 7f8d86a30700 1 -- 172.21.15.195:0/3832697292 shutdown_connections 2023-10-12T22:39:21.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.007+0000 7f8d86a30700 1 --2- 172.21.15.195:0/3832697292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d80060220 0x7f8d800fb550 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:21.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.007+0000 7f8d86a30700 1 -- 172.21.15.195:0/3832697292 >> 172.21.15.195:0/3832697292 conn(0x7f8d800f0b70 msgr2=0x7f8d800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:21.009 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.007+0000 7f8d86a30700 1 -- 172.21.15.195:0/3832697292 shutdown_connections 2023-10-12T22:39:21.010 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.007+0000 7f8d86a30700 1 -- 172.21.15.195:0/3832697292 wait complete. 2023-10-12T22:39:21.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.008+0000 7f8d86a30700 1 Processor -- start 2023-10-12T22:39:21.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.008+0000 7f8d86a30700 1 -- start start 2023-10-12T22:39:21.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.008+0000 7f8d86a30700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d80060220 0x7f8d801088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:21.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.008+0000 7f8d86a30700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d80108db0 con 0x7f8d80060220 2023-10-12T22:39:21.012 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.008+0000 7f8d7ffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d80060220 0x7f8d801088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:21.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.008+0000 7f8d7ffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d80060220 0x7f8d801088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51928/0 (socket says 172.21.15.195:51928) 2023-10-12T22:39:21.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.008+0000 7f8d7ffff700 1 -- 172.21.15.195:0/1950890314 learned_addr learned my addr 172.21.15.195:0/1950890314 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:21.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.009+0000 7f8d7ffff700 1 -- 172.21.15.195:0/1950890314 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70005040 con 0x7f8d80060220 2023-10-12T22:39:21.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.009+0000 7f8d7ffff700 1 --2- 172.21.15.195:0/1950890314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d80060220 0x7f8d801088c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f8d70013510 tx=0x7f8d7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:21.013 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.009+0000 7f8d7cff9700 1 -- 172.21.15.195:0/1950890314 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d70009320 con 0x7f8d80060220 2023-10-12T22:39:21.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.009+0000 7f8d86a30700 1 -- 172.21.15.195:0/1950890314 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d80108fb0 con 0x7f8d80060220 2023-10-12T22:39:21.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.009+0000 7f8d7cff9700 1 -- 172.21.15.195:0/1950890314 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d70013f00 con 0x7f8d80060220 2023-10-12T22:39:21.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.009+0000 7f8d7cff9700 1 -- 172.21.15.195:0/1950890314 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d70014dc0 con 0x7f8d80060220 2023-10-12T22:39:21.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.009+0000 7f8d86a30700 1 -- 172.21.15.195:0/1950890314 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d80109440 con 0x7f8d80060220 2023-10-12T22:39:21.014 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.010+0000 7f8d7cff9700 1 -- 172.21.15.195:0/1950890314 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d7001d3d0 con 0x7f8d80060220 2023-10-12T22:39:21.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.010+0000 7f8d86a30700 1 -- 172.21.15.195:0/1950890314 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d80045ad0 con 0x7f8d80060220 2023-10-12T22:39:21.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.010+0000 7f8d7cff9700 1 --2- 172.21.15.195:0/1950890314 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d6802ffb0 0x7f8d68032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:21.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.010+0000 7f8d7cff9700 1 -- 172.21.15.195:0/1950890314 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d70041100 con 0x7f8d80060220 2023-10-12T22:39:21.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.010+0000 7f8d7f7fe700 1 --2- 172.21.15.195:0/1950890314 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d6802ffb0 0x7f8d68032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:21.015 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.011+0000 7f8d7f7fe700 1 --2- 172.21.15.195:0/1950890314 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d6802ffb0 0x7f8d68032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f8d740096a0 tx=0x7f8d74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:21.016 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.015+0000 7f8d7cff9700 1 -- 172.21.15.195:0/1950890314 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d700126f0 con 0x7f8d80060220 2023-10-12T22:39:21.224 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.223+0000 7f8d86a30700 1 -- 172.21.15.195:0/1950890314 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d8010a290 con 0x7f8d80060220 2023-10-12T22:39:21.225 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.224+0000 7f8d7cff9700 1 -- 172.21.15.195:0/1950890314 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8d70012330 con 0x7f8d80060220 2023-10-12T22:39:21.226 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:21.226 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:21.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.226+0000 7f8d86a30700 1 -- 172.21.15.195:0/1950890314 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d6802ffb0 msgr2=0x7f8d68032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:21.227 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.226+0000 7f8d86a30700 1 --2- 172.21.15.195:0/1950890314 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d6802ffb0 0x7f8d68032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f8d740096a0 tx=0x7f8d74006b40).stop 2023-10-12T22:39:21.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.226+0000 7f8d86a30700 1 -- 172.21.15.195:0/1950890314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d80060220 msgr2=0x7f8d801088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:21.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.226+0000 7f8d86a30700 1 --2- 172.21.15.195:0/1950890314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d80060220 0x7f8d801088c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f8d70013510 tx=0x7f8d7000b040).stop 2023-10-12T22:39:21.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.226+0000 7f8d86a30700 1 -- 172.21.15.195:0/1950890314 shutdown_connections 2023-10-12T22:39:21.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.226+0000 7f8d86a30700 1 --2- 172.21.15.195:0/1950890314 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f8d6802ffb0 0x7f8d68032470 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:21.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.226+0000 7f8d86a30700 1 --2- 172.21.15.195:0/1950890314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8d80060220 0x7f8d801088c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:21.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.226+0000 7f8d86a30700 1 -- 172.21.15.195:0/1950890314 >> 172.21.15.195:0/1950890314 conn(0x7f8d800f0b70 msgr2=0x7f8d800f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:21.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.226+0000 7f8d86a30700 1 -- 172.21.15.195:0/1950890314 shutdown_connections 2023-10-12T22:39:21.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:21.226+0000 7f8d86a30700 1 -- 172.21.15.195:0/1950890314 wait complete. 2023-10-12T22:39:21.229 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:22.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:20.872052+0000 mgr.smithi131.efktma (mgr.14162) 245 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:22.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:21.225506+0000 mon.smithi131 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.195:0/1950890314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:22.646 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:22.647 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:23.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:22.939663+0000 mon.smithi131 (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/mirror_snapshot_schedule"}]: dispatch 2023-10-12T22:39:23.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:23 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:22.944356+0000 mon.smithi131 (mon.0) 303 : audit [INF] from='mgr.14162 172.21.15.131:0/1229472172' entity='mgr.smithi131.efktma' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.efktma/trash_purge_schedule"}]: dispatch 2023-10-12T22:39:23.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.740+0000 7f18709e8700 1 -- 172.21.15.195:0/1462565853 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f186c060180 msgr2=0x7f186c0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:23.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.740+0000 7f18709e8700 1 --2- 172.21.15.195:0/1462565853 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f186c060180 0x7f186c0f8f80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f1854009960 tx=0x7f1854004d10).stop 2023-10-12T22:39:23.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.740+0000 7f18709e8700 1 -- 172.21.15.195:0/1462565853 shutdown_connections 2023-10-12T22:39:23.742 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.740+0000 7f18709e8700 1 --2- 172.21.15.195:0/1462565853 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f186c060180 0x7f186c0f8f80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:23.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.740+0000 7f18709e8700 1 -- 172.21.15.195:0/1462565853 >> 172.21.15.195:0/1462565853 conn(0x7f186c0f09a0 msgr2=0x7f186c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:23.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.740+0000 7f18709e8700 1 -- 172.21.15.195:0/1462565853 shutdown_connections 2023-10-12T22:39:23.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.740+0000 7f18709e8700 1 -- 172.21.15.195:0/1462565853 wait complete. 2023-10-12T22:39:23.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.740+0000 7f18709e8700 1 Processor -- start 2023-10-12T22:39:23.743 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.741+0000 7f18709e8700 1 -- start start 2023-10-12T22:39:23.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.741+0000 7f18709e8700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f186c060180 0x7f186c1086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:23.744 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.741+0000 7f18709e8700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186c108bc0 con 0x7f186c060180 2023-10-12T22:39:23.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.741+0000 7f186b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f186c060180 0x7f186c1086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:23.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.741+0000 7f186b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f186c060180 0x7f186c1086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51952/0 (socket says 172.21.15.195:51952) 2023-10-12T22:39:23.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.741+0000 7f186b7fe700 1 -- 172.21.15.195:0/2311353745 learned_addr learned my addr 172.21.15.195:0/2311353745 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:23.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.741+0000 7f186b7fe700 1 -- 172.21.15.195:0/2311353745 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1854009640 con 0x7f186c060180 2023-10-12T22:39:23.746 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.742+0000 7f186b7fe700 1 --2- 172.21.15.195:0/2311353745 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f186c060180 0x7f186c1086d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f1854004b70 tx=0x7f1854004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:23.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.742+0000 7f1863fff700 1 -- 172.21.15.195:0/2311353745 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1854018070 con 0x7f186c060180 2023-10-12T22:39:23.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.742+0000 7f1863fff700 1 -- 172.21.15.195:0/2311353745 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f185400b870 con 0x7f186c060180 2023-10-12T22:39:23.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.742+0000 7f18709e8700 1 -- 172.21.15.195:0/2311353745 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186c108dc0 con 0x7f186c060180 2023-10-12T22:39:23.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.742+0000 7f1863fff700 1 -- 172.21.15.195:0/2311353745 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f185401c560 con 0x7f186c060180 2023-10-12T22:39:23.747 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.742+0000 7f18709e8700 1 -- 172.21.15.195:0/2311353745 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186c109250 con 0x7f186c060180 2023-10-12T22:39:23.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.743+0000 7f1863fff700 1 -- 172.21.15.195:0/2311353745 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1854013630 con 0x7f186c060180 2023-10-12T22:39:23.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.743+0000 7f1863fff700 1 --2- 172.21.15.195:0/2311353745 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f185802ff60 0x7f1858032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:23.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.743+0000 7f1863fff700 1 -- 172.21.15.195:0/2311353745 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18540411d0 con 0x7f186c060180 2023-10-12T22:39:23.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.743+0000 7f18709e8700 1 -- 172.21.15.195:0/2311353745 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186c059100 con 0x7f186c060180 2023-10-12T22:39:23.748 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.743+0000 7f186affd700 1 --2- 172.21.15.195:0/2311353745 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f185802ff60 0x7f1858032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:23.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.743+0000 7f186affd700 1 --2- 172.21.15.195:0/2311353745 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f185802ff60 0x7f1858032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f185c0096a0 tx=0x7f185c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:23.749 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.748+0000 7f1863fff700 1 -- 172.21.15.195:0/2311353745 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1854004810 con 0x7f186c060180 2023-10-12T22:39:23.957 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.956+0000 7f18709e8700 1 -- 172.21.15.195:0/2311353745 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f186c045ad0 con 0x7f186c060180 2023-10-12T22:39:23.958 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.957+0000 7f1863fff700 1 -- 172.21.15.195:0/2311353745 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1854012380 con 0x7f186c060180 2023-10-12T22:39:23.958 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:23.958 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:23.960 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.958+0000 7f18709e8700 1 -- 172.21.15.195:0/2311353745 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f185802ff60 msgr2=0x7f1858032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:23.960 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.958+0000 7f18709e8700 1 --2- 172.21.15.195:0/2311353745 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f185802ff60 0x7f1858032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f185c0096a0 tx=0x7f185c006b40).stop 2023-10-12T22:39:23.960 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.958+0000 7f18709e8700 1 -- 172.21.15.195:0/2311353745 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f186c060180 msgr2=0x7f186c1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:23.960 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.958+0000 7f18709e8700 1 --2- 172.21.15.195:0/2311353745 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f186c060180 0x7f186c1086d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f1854004b70 tx=0x7f1854004d10).stop 2023-10-12T22:39:23.960 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.958+0000 7f18709e8700 1 -- 172.21.15.195:0/2311353745 shutdown_connections 2023-10-12T22:39:23.961 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.958+0000 7f18709e8700 1 --2- 172.21.15.195:0/2311353745 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f185802ff60 0x7f1858032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:23.961 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.958+0000 7f18709e8700 1 --2- 172.21.15.195:0/2311353745 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f186c060180 0x7f186c1086d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:23.961 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.958+0000 7f18709e8700 1 -- 172.21.15.195:0/2311353745 >> 172.21.15.195:0/2311353745 conn(0x7f186c0f09a0 msgr2=0x7f186c0f1490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:23.961 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.959+0000 7f18709e8700 1 -- 172.21.15.195:0/2311353745 shutdown_connections 2023-10-12T22:39:23.961 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:23.959+0000 7f18709e8700 1 -- 172.21.15.195:0/2311353745 wait complete. 2023-10-12T22:39:23.962 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:24.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:22.872440+0000 mgr.smithi131.efktma (mgr.14162) 246 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:24.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:24 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:23.958035+0000 mon.smithi131 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.195:0/2311353745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:25.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:25.295 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:26.406 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.403+0000 7f0749bab700 1 -- 172.21.15.195:0/3000902273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0744060220 msgr2=0x7f07440f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:26.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.403+0000 7f0749bab700 1 --2- 172.21.15.195:0/3000902273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0744060220 0x7f07440f9150 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f0734009fb0 tx=0x7f0734009b30).stop 2023-10-12T22:39:26.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.404+0000 7f0749bab700 1 -- 172.21.15.195:0/3000902273 shutdown_connections 2023-10-12T22:39:26.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.404+0000 7f0749bab700 1 --2- 172.21.15.195:0/3000902273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0744060220 0x7f07440f9150 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:26.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.404+0000 7f0749bab700 1 -- 172.21.15.195:0/3000902273 >> 172.21.15.195:0/3000902273 conn(0x7f07440f0b70 msgr2=0x7f07440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:26.407 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.404+0000 7f0749bab700 1 -- 172.21.15.195:0/3000902273 shutdown_connections 2023-10-12T22:39:26.408 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.404+0000 7f0749bab700 1 -- 172.21.15.195:0/3000902273 wait complete. 2023-10-12T22:39:26.408 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.404+0000 7f0749bab700 1 Processor -- start 2023-10-12T22:39:26.408 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.404+0000 7f0749bab700 1 -- start start 2023-10-12T22:39:26.408 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.404+0000 7f0749bab700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0744060220 0x7f07441066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:26.408 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.404+0000 7f0749bab700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0744106bd0 con 0x7f0744060220 2023-10-12T22:39:26.409 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.405+0000 7f07437fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0744060220 0x7f07441066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:26.409 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.405+0000 7f07437fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0744060220 0x7f07441066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51980/0 (socket says 172.21.15.195:51980) 2023-10-12T22:39:26.409 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.405+0000 7f07437fe700 1 -- 172.21.15.195:0/3958996217 learned_addr learned my addr 172.21.15.195:0/3958996217 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:26.409 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.405+0000 7f07437fe700 1 -- 172.21.15.195:0/3958996217 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0734005040 con 0x7f0744060220 2023-10-12T22:39:26.409 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.405+0000 7f07437fe700 1 --2- 172.21.15.195:0/3958996217 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0744060220 0x7f07441066e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f0734013510 tx=0x7f073400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:26.410 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.405+0000 7f073bfff700 1 -- 172.21.15.195:0/3958996217 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0734009320 con 0x7f0744060220 2023-10-12T22:39:26.410 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.405+0000 7f0749bab700 1 -- 172.21.15.195:0/3958996217 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0744106dd0 con 0x7f0744060220 2023-10-12T22:39:26.410 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.405+0000 7f073bfff700 1 -- 172.21.15.195:0/3958996217 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0734013f00 con 0x7f0744060220 2023-10-12T22:39:26.410 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.405+0000 7f0749bab700 1 -- 172.21.15.195:0/3958996217 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0744107260 con 0x7f0744060220 2023-10-12T22:39:26.410 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.405+0000 7f073bfff700 1 -- 172.21.15.195:0/3958996217 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0734014de0 con 0x7f0744060220 2023-10-12T22:39:26.412 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.406+0000 7f073bfff700 1 -- 172.21.15.195:0/3958996217 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0734014420 con 0x7f0744060220 2023-10-12T22:39:26.412 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.406+0000 7f0749bab700 1 -- 172.21.15.195:0/3958996217 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0744045ad0 con 0x7f0744060220 2023-10-12T22:39:26.412 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.406+0000 7f073bfff700 1 --2- 172.21.15.195:0/3958996217 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f073002ff60 0x7f0730032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:26.412 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.406+0000 7f073bfff700 1 -- 172.21.15.195:0/3958996217 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07340410a0 con 0x7f0744060220 2023-10-12T22:39:26.412 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.407+0000 7f0742ffd700 1 --2- 172.21.15.195:0/3958996217 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f073002ff60 0x7f0730032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:26.413 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.407+0000 7f0742ffd700 1 --2- 172.21.15.195:0/3958996217 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f073002ff60 0x7f0730032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f072c0096a0 tx=0x7f072c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:26.413 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.411+0000 7f073bfff700 1 -- 172.21.15.195:0/3958996217 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07340095f0 con 0x7f0744060220 2023-10-12T22:39:26.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:26 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:24.872753+0000 mgr.smithi131.efktma (mgr.14162) 247 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:26.615 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.614+0000 7f0749bab700 1 -- 172.21.15.195:0/3958996217 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0744108110 con 0x7f0744060220 2023-10-12T22:39:26.616 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.614+0000 7f073bfff700 1 -- 172.21.15.195:0/3958996217 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f073402b2d0 con 0x7f0744060220 2023-10-12T22:39:26.616 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:26.616 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:26.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.616+0000 7f0749bab700 1 -- 172.21.15.195:0/3958996217 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f073002ff60 msgr2=0x7f0730032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:26.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.616+0000 7f0749bab700 1 --2- 172.21.15.195:0/3958996217 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f073002ff60 0x7f0730032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f072c0096a0 tx=0x7f072c006b40).stop 2023-10-12T22:39:26.617 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.616+0000 7f0749bab700 1 -- 172.21.15.195:0/3958996217 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0744060220 msgr2=0x7f07441066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:26.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.616+0000 7f0749bab700 1 --2- 172.21.15.195:0/3958996217 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0744060220 0x7f07441066e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f0734013510 tx=0x7f073400b040).stop 2023-10-12T22:39:26.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.616+0000 7f0749bab700 1 -- 172.21.15.195:0/3958996217 shutdown_connections 2023-10-12T22:39:26.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.616+0000 7f0749bab700 1 --2- 172.21.15.195:0/3958996217 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f073002ff60 0x7f0730032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:26.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.616+0000 7f0749bab700 1 --2- 172.21.15.195:0/3958996217 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0744060220 0x7f07441066e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:26.618 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.616+0000 7f0749bab700 1 -- 172.21.15.195:0/3958996217 >> 172.21.15.195:0/3958996217 conn(0x7f07440f0b70 msgr2=0x7f07440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:26.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.616+0000 7f0749bab700 1 -- 172.21.15.195:0/3958996217 shutdown_connections 2023-10-12T22:39:26.619 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:26.616+0000 7f0749bab700 1 -- 172.21.15.195:0/3958996217 wait complete. 2023-10-12T22:39:26.619 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:27.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:27 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:26.615828+0000 mon.smithi131 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.195:0/3958996217' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:27.987 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:27.988 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:28.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:28 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:26.873095+0000 mgr.smithi131.efktma (mgr.14162) 248 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:29.131 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.130+0000 7fbf36c29700 1 -- 172.21.15.195:0/963418967 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf300f5410 msgr2=0x7fbf300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:29.132 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.130+0000 7fbf36c29700 1 --2- 172.21.15.195:0/963418967 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf300f5410 0x7fbf300f5830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fbf24009fb0 tx=0x7fbf24009b30).stop 2023-10-12T22:39:29.132 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.130+0000 7fbf36c29700 1 -- 172.21.15.195:0/963418967 shutdown_connections 2023-10-12T22:39:29.132 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.130+0000 7fbf36c29700 1 --2- 172.21.15.195:0/963418967 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf300f5410 0x7fbf300f5830 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:29.132 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.130+0000 7fbf36c29700 1 -- 172.21.15.195:0/963418967 >> 172.21.15.195:0/963418967 conn(0x7fbf300f0b70 msgr2=0x7fbf300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:29.133 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.130+0000 7fbf36c29700 1 -- 172.21.15.195:0/963418967 shutdown_connections 2023-10-12T22:39:29.133 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.130+0000 7fbf36c29700 1 -- 172.21.15.195:0/963418967 wait complete. 2023-10-12T22:39:29.133 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.130+0000 7fbf36c29700 1 Processor -- start 2023-10-12T22:39:29.133 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.130+0000 7fbf36c29700 1 -- start start 2023-10-12T22:39:29.135 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.131+0000 7fbf36c29700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf300f5410 0x7fbf30108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:29.136 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.131+0000 7fbf36c29700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf30108e10 con 0x7fbf300f5410 2023-10-12T22:39:29.136 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.131+0000 7fbf349c5700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf300f5410 0x7fbf30108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:29.136 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.131+0000 7fbf349c5700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf300f5410 0x7fbf30108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:55990/0 (socket says 172.21.15.195:55990) 2023-10-12T22:39:29.136 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.131+0000 7fbf349c5700 1 -- 172.21.15.195:0/886455194 learned_addr learned my addr 172.21.15.195:0/886455194 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:29.137 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.131+0000 7fbf349c5700 1 -- 172.21.15.195:0/886455194 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf24005040 con 0x7fbf300f5410 2023-10-12T22:39:29.137 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.131+0000 7fbf349c5700 1 --2- 172.21.15.195:0/886455194 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf300f5410 0x7fbf30108920 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fbf24013510 tx=0x7fbf2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:29.137 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.132+0000 7fbf2d7fa700 1 -- 172.21.15.195:0/886455194 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf24009320 con 0x7fbf300f5410 2023-10-12T22:39:29.137 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.132+0000 7fbf36c29700 1 -- 172.21.15.195:0/886455194 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf30109010 con 0x7fbf300f5410 2023-10-12T22:39:29.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.132+0000 7fbf2d7fa700 1 -- 172.21.15.195:0/886455194 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf24013f00 con 0x7fbf300f5410 2023-10-12T22:39:29.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.132+0000 7fbf2d7fa700 1 -- 172.21.15.195:0/886455194 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf24014d90 con 0x7fbf300f5410 2023-10-12T22:39:29.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.132+0000 7fbf36c29700 1 -- 172.21.15.195:0/886455194 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf301094a0 con 0x7fbf300f5410 2023-10-12T22:39:29.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.132+0000 7fbf2d7fa700 1 -- 172.21.15.195:0/886455194 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf240263a0 con 0x7fbf300f5410 2023-10-12T22:39:29.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.133+0000 7fbf2d7fa700 1 --2- 172.21.15.195:0/886455194 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbf2002ff60 0x7fbf20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:29.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.133+0000 7fbf2d7fa700 1 -- 172.21.15.195:0/886455194 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf24041f90 con 0x7fbf300f5410 2023-10-12T22:39:29.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.133+0000 7fbf36c29700 1 -- 172.21.15.195:0/886455194 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf30059100 con 0x7fbf300f5410 2023-10-12T22:39:29.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.133+0000 7fbf2ffff700 1 --2- 172.21.15.195:0/886455194 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbf2002ff60 0x7fbf20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:29.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.133+0000 7fbf2ffff700 1 --2- 172.21.15.195:0/886455194 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbf2002ff60 0x7fbf20032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fbf1c0096a0 tx=0x7fbf1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:29.140 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.137+0000 7fbf2d7fa700 1 -- 172.21.15.195:0/886455194 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf24012920 con 0x7fbf300f5410 2023-10-12T22:39:29.346 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.345+0000 7fbf36c29700 1 -- 172.21.15.195:0/886455194 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf300fae60 con 0x7fbf300f5410 2023-10-12T22:39:29.347 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.346+0000 7fbf2d7fa700 1 -- 172.21.15.195:0/886455194 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbf2402b020 con 0x7fbf300f5410 2023-10-12T22:39:29.347 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:29.348 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:29.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.347+0000 7fbf36c29700 1 -- 172.21.15.195:0/886455194 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbf2002ff60 msgr2=0x7fbf20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:29.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.347+0000 7fbf36c29700 1 --2- 172.21.15.195:0/886455194 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbf2002ff60 0x7fbf20032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fbf1c0096a0 tx=0x7fbf1c006b40).stop 2023-10-12T22:39:29.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.347+0000 7fbf36c29700 1 -- 172.21.15.195:0/886455194 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf300f5410 msgr2=0x7fbf30108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:29.349 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.347+0000 7fbf36c29700 1 --2- 172.21.15.195:0/886455194 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf300f5410 0x7fbf30108920 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fbf24013510 tx=0x7fbf2400b040).stop 2023-10-12T22:39:29.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.347+0000 7fbf36c29700 1 -- 172.21.15.195:0/886455194 shutdown_connections 2023-10-12T22:39:29.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.347+0000 7fbf36c29700 1 --2- 172.21.15.195:0/886455194 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fbf2002ff60 0x7fbf20032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:29.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.347+0000 7fbf36c29700 1 --2- 172.21.15.195:0/886455194 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf300f5410 0x7fbf30108920 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:29.350 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.347+0000 7fbf36c29700 1 -- 172.21.15.195:0/886455194 >> 172.21.15.195:0/886455194 conn(0x7fbf300f0b70 msgr2=0x7fbf3005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:29.351 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.347+0000 7fbf36c29700 1 -- 172.21.15.195:0/886455194 shutdown_connections 2023-10-12T22:39:29.351 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:29.347+0000 7fbf36c29700 1 -- 172.21.15.195:0/886455194 wait complete. 2023-10-12T22:39:29.351 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:29.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:39:28.276663+0000 mgr.smithi131.efktma (mgr.14162) 249 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi131 ... 2023-10-12T22:39:29.491 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Verifying port 9100 ... 2023-10-12T22:39:29.491 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Non-zero exit code 1 from systemctl start ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@node-exporter.smithi131 2023-10-12T22:39:29.492 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: systemctl: stderr Job for ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@node-exporter.smithi131.service failed because a timeout was exceeded. 2023-10-12T22:39:29.492 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: systemctl: stderr See "systemctl status ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@node-exporter.smithi131.service" and "journalctl -xe" for details. 2023-10-12T22:39:29.492 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Traceback (most recent call last): 2023-10-12T22:39:29.493 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-10-12T22:39:29.493 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: main() 2023-10-12T22:39:29.493 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-10-12T22:39:29.494 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: r = ctx.func(ctx) 2023-10-12T22:39:29.494 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-10-12T22:39:29.494 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: return func(ctx) 2023-10-12T22:39:29.495 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-10-12T22:39:29.495 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ports=daemon_ports) 2023-10-12T22:39:29.495 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-10-12T22:39:29.496 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: c, osd_fsid=osd_fsid, ports=ports) 2023-10-12T22:39:29.496 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-10-12T22:39:29.496 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-10-12T22:39:29.496 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-10-12T22:39:29.496 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-10-12T22:39:29.497 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: RuntimeError: Failed command: systemctl start ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@node-exporter.smithi131 2023-10-12T22:39:29.497 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Traceback (most recent call last): 2023-10-12T22:39:29.497 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-10-12T22:39:29.497 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: yield (conn, connr) 2023-10-12T22:39:29.497 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-10-12T22:39:29.498 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: code, '\n'.join(err))) 2023-10-12T22:39:29.498 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi131 ... 2023-10-12T22:39:29.498 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Verifying port 9100 ... 2023-10-12T22:39:29.498 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Non-zero exit code 1 from systemctl start ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@node-exporter.smithi131 2023-10-12T22:39:29.499 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: systemctl: stderr Job for ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@node-exporter.smithi131.service failed because a timeout was exceeded. 2023-10-12T22:39:29.499 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: systemctl: stderr See "systemctl status ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@node-exporter.smithi131.service" and "journalctl -xe" for details. 2023-10-12T22:39:29.499 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: Traceback (most recent call last): 2023-10-12T22:39:29.499 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-10-12T22:39:29.499 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: main() 2023-10-12T22:39:29.500 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-10-12T22:39:29.500 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: r = ctx.func(ctx) 2023-10-12T22:39:29.500 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-10-12T22:39:29.500 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: return func(ctx) 2023-10-12T22:39:29.501 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-10-12T22:39:29.501 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: ports=daemon_ports) 2023-10-12T22:39:29.501 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-10-12T22:39:29.501 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: c, osd_fsid=osd_fsid, ports=ports) 2023-10-12T22:39:29.501 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-10-12T22:39:29.502 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-10-12T22:39:29.502 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: File "/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-10-12T22:39:29.502 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-10-12T22:39:29.503 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: RuntimeError: Failed command: systemctl start ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@node-exporter.smithi131 2023-10-12T22:39:29.503 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:29 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cephadm 2023-10-12T22:39:28.281991+0000 mgr.smithi131.efktma (mgr.14162) 250 : cephadm [INF] Deploying daemon prometheus.smithi131 on smithi131 2023-10-12T22:39:30.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:30 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:28.873367+0000 mgr.smithi131.efktma (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:30.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:30 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:29.347209+0000 mon.smithi131 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.195:0/886455194' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:30.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:30.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:31.900 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.897+0000 7fd532117700 1 -- 172.21.15.195:0/2303506804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd52c0f46f0 msgr2=0x7fd52c0f4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:31.900 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.897+0000 7fd532117700 1 --2- 172.21.15.195:0/2303506804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd52c0f46f0 0x7fd52c0f4b10 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fd51c009fb0 tx=0x7fd51c009b30).stop 2023-10-12T22:39:31.900 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.897+0000 7fd532117700 1 -- 172.21.15.195:0/2303506804 shutdown_connections 2023-10-12T22:39:31.900 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.897+0000 7fd532117700 1 --2- 172.21.15.195:0/2303506804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd52c0f46f0 0x7fd52c0f4b10 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:31.901 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.897+0000 7fd532117700 1 -- 172.21.15.195:0/2303506804 >> 172.21.15.195:0/2303506804 conn(0x7fd52c0f0210 msgr2=0x7fd52c0f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:31.901 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.898+0000 7fd532117700 1 -- 172.21.15.195:0/2303506804 shutdown_connections 2023-10-12T22:39:31.901 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.898+0000 7fd532117700 1 -- 172.21.15.195:0/2303506804 wait complete. 2023-10-12T22:39:31.901 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.898+0000 7fd532117700 1 Processor -- start 2023-10-12T22:39:31.901 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.898+0000 7fd532117700 1 -- start start 2023-10-12T22:39:31.902 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.899+0000 7fd532117700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd52c0f46f0 0x7fd52c106660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:31.902 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.899+0000 7fd532117700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c106b50 con 0x7fd52c0f46f0 2023-10-12T22:39:31.903 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.899+0000 7fd52b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd52c0f46f0 0x7fd52c106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:31.904 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.899+0000 7fd52b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd52c0f46f0 0x7fd52c106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:56000/0 (socket says 172.21.15.195:56000) 2023-10-12T22:39:31.904 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.899+0000 7fd52b7fe700 1 -- 172.21.15.195:0/2868408075 learned_addr learned my addr 172.21.15.195:0/2868408075 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:31.904 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.899+0000 7fd52b7fe700 1 -- 172.21.15.195:0/2868408075 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c005040 con 0x7fd52c0f46f0 2023-10-12T22:39:31.904 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.899+0000 7fd52b7fe700 1 --2- 172.21.15.195:0/2868408075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd52c0f46f0 0x7fd52c106660 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd51c013510 tx=0x7fd51c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:31.904 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.900+0000 7fd513fff700 1 -- 172.21.15.195:0/2868408075 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd51c009320 con 0x7fd52c0f46f0 2023-10-12T22:39:31.905 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.900+0000 7fd513fff700 1 -- 172.21.15.195:0/2868408075 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd51c013f00 con 0x7fd52c0f46f0 2023-10-12T22:39:31.905 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.900+0000 7fd513fff700 1 -- 172.21.15.195:0/2868408075 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd51c014d90 con 0x7fd52c0f46f0 2023-10-12T22:39:31.905 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.900+0000 7fd532117700 1 -- 172.21.15.195:0/2868408075 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52c106d50 con 0x7fd52c0f46f0 2023-10-12T22:39:31.905 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.900+0000 7fd532117700 1 -- 172.21.15.195:0/2868408075 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52c1071e0 con 0x7fd52c0f46f0 2023-10-12T22:39:31.905 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.901+0000 7fd513fff700 1 -- 172.21.15.195:0/2868408075 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd51c0095f0 con 0x7fd52c0f46f0 2023-10-12T22:39:31.906 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.901+0000 7fd513fff700 1 --2- 172.21.15.195:0/2868408075 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd51402ffb0 0x7fd514032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:31.906 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.901+0000 7fd532117700 1 -- 172.21.15.195:0/2868408075 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd52c045ad0 con 0x7fd52c0f46f0 2023-10-12T22:39:31.906 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.901+0000 7fd513fff700 1 -- 172.21.15.195:0/2868408075 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd51c040ff0 con 0x7fd52c0f46f0 2023-10-12T22:39:31.906 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.901+0000 7fd52affd700 1 --2- 172.21.15.195:0/2868408075 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd51402ffb0 0x7fd514032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:31.906 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.902+0000 7fd52affd700 1 --2- 172.21.15.195:0/2868408075 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd51402ffb0 0x7fd514032470 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fd5200096a0 tx=0x7fd520006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:31.907 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:31.906+0000 7fd513fff700 1 -- 172.21.15.195:0/2868408075 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd51c014420 con 0x7fd52c0f46f0 2023-10-12T22:39:32.114 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:32.113+0000 7fd532117700 1 -- 172.21.15.195:0/2868408075 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd52c0fbef0 con 0x7fd52c0f46f0 2023-10-12T22:39:32.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:32.114+0000 7fd513fff700 1 -- 172.21.15.195:0/2868408075 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd51c02b020 con 0x7fd52c0f46f0 2023-10-12T22:39:32.115 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:32.116 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:32.116 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:32.115+0000 7fd532117700 1 -- 172.21.15.195:0/2868408075 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd51402ffb0 msgr2=0x7fd514032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:32.117 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:32.115+0000 7fd532117700 1 --2- 172.21.15.195:0/2868408075 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd51402ffb0 0x7fd514032470 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fd5200096a0 tx=0x7fd520006b40).stop 2023-10-12T22:39:32.117 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:32.115+0000 7fd532117700 1 -- 172.21.15.195:0/2868408075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd52c0f46f0 msgr2=0x7fd52c106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:32.117 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:32.115+0000 7fd532117700 1 --2- 172.21.15.195:0/2868408075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd52c0f46f0 0x7fd52c106660 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd51c013510 tx=0x7fd51c00b040).stop 2023-10-12T22:39:32.117 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:32.115+0000 7fd532117700 1 -- 172.21.15.195:0/2868408075 shutdown_connections 2023-10-12T22:39:32.117 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:32.115+0000 7fd532117700 1 --2- 172.21.15.195:0/2868408075 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd51402ffb0 0x7fd514032470 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:32.118 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:32.115+0000 7fd532117700 1 --2- 172.21.15.195:0/2868408075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd52c0f46f0 0x7fd52c106660 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:32.118 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:32.115+0000 7fd532117700 1 -- 172.21.15.195:0/2868408075 >> 172.21.15.195:0/2868408075 conn(0x7fd52c0f0210 msgr2=0x7fd52c0f0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:32.118 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:32.116+0000 7fd532117700 1 -- 172.21.15.195:0/2868408075 shutdown_connections 2023-10-12T22:39:32.118 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:32.116+0000 7fd532117700 1 -- 172.21.15.195:0/2868408075 wait complete. 2023-10-12T22:39:32.119 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:32.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:30.873648+0000 mgr.smithi131.efktma (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:32.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:32 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:32.115372+0000 mon.smithi131 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.195:0/2868408075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:33.519 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:33.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:34.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:34 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:32.874025+0000 mgr.smithi131.efktma (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:34.700 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.698+0000 7fd697b6c700 1 -- 172.21.15.195:0/1054007739 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6900f5410 msgr2=0x7fd6900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:34.701 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.698+0000 7fd697b6c700 1 --2- 172.21.15.195:0/1054007739 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6900f5410 0x7fd6900f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fd684009fb0 tx=0x7fd684009b30).stop 2023-10-12T22:39:34.701 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.698+0000 7fd697b6c700 1 -- 172.21.15.195:0/1054007739 shutdown_connections 2023-10-12T22:39:34.701 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.698+0000 7fd697b6c700 1 --2- 172.21.15.195:0/1054007739 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6900f5410 0x7fd6900f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:34.702 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.698+0000 7fd697b6c700 1 -- 172.21.15.195:0/1054007739 >> 172.21.15.195:0/1054007739 conn(0x7fd6900f0b70 msgr2=0x7fd6900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:34.702 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.698+0000 7fd697b6c700 1 -- 172.21.15.195:0/1054007739 shutdown_connections 2023-10-12T22:39:34.702 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.698+0000 7fd697b6c700 1 -- 172.21.15.195:0/1054007739 wait complete. 2023-10-12T22:39:34.703 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.699+0000 7fd697b6c700 1 Processor -- start 2023-10-12T22:39:34.703 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.699+0000 7fd697b6c700 1 -- start start 2023-10-12T22:39:34.706 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.699+0000 7fd697b6c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6900f5410 0x7fd690109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:34.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.699+0000 7fd697b6c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd690060220 con 0x7fd6900f5410 2023-10-12T22:39:34.707 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.699+0000 7fd695908700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6900f5410 0x7fd690109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:34.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.699+0000 7fd695908700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6900f5410 0x7fd690109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:56014/0 (socket says 172.21.15.195:56014) 2023-10-12T22:39:34.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.699+0000 7fd695908700 1 -- 172.21.15.195:0/1432586929 learned_addr learned my addr 172.21.15.195:0/1432586929 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:34.708 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.700+0000 7fd695908700 1 -- 172.21.15.195:0/1432586929 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd684005040 con 0x7fd6900f5410 2023-10-12T22:39:34.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.700+0000 7fd695908700 1 --2- 172.21.15.195:0/1432586929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6900f5410 0x7fd690109450 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fd684013510 tx=0x7fd68400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:34.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.700+0000 7fd6827fc700 1 -- 172.21.15.195:0/1432586929 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd684009320 con 0x7fd6900f5410 2023-10-12T22:39:34.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.700+0000 7fd697b6c700 1 -- 172.21.15.195:0/1432586929 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd690060420 con 0x7fd6900f5410 2023-10-12T22:39:34.709 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.700+0000 7fd6827fc700 1 -- 172.21.15.195:0/1432586929 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd684013f00 con 0x7fd6900f5410 2023-10-12T22:39:34.710 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.700+0000 7fd6827fc700 1 -- 172.21.15.195:0/1432586929 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd684014d90 con 0x7fd6900f5410 2023-10-12T22:39:34.710 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.700+0000 7fd697b6c700 1 -- 172.21.15.195:0/1432586929 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd690109b00 con 0x7fd6900f5410 2023-10-12T22:39:34.710 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.701+0000 7fd6827fc700 1 -- 172.21.15.195:0/1432586929 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd6840095f0 con 0x7fd6900f5410 2023-10-12T22:39:34.711 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.701+0000 7fd697b6c700 1 -- 172.21.15.195:0/1432586929 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd690059100 con 0x7fd6900f5410 2023-10-12T22:39:34.711 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.701+0000 7fd6827fc700 1 --2- 172.21.15.195:0/1432586929 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd67c02ff60 0x7fd67c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:34.711 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.701+0000 7fd6827fc700 1 -- 172.21.15.195:0/1432586929 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd684041000 con 0x7fd6900f5410 2023-10-12T22:39:34.712 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.701+0000 7fd695107700 1 --2- 172.21.15.195:0/1432586929 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd67c02ff60 0x7fd67c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:34.712 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.702+0000 7fd695107700 1 --2- 172.21.15.195:0/1432586929 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd67c02ff60 0x7fd67c032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fd68c0096a0 tx=0x7fd68c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:34.713 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.706+0000 7fd6827fc700 1 -- 172.21.15.195:0/1432586929 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd684014420 con 0x7fd6900f5410 2023-10-12T22:39:34.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.913+0000 7fd697b6c700 1 -- 172.21.15.195:0/1432586929 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd690045ad0 con 0x7fd6900f5410 2023-10-12T22:39:34.915 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.913+0000 7fd6827fc700 1 -- 172.21.15.195:0/1432586929 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd68401d7f0 con 0x7fd6900f5410 2023-10-12T22:39:34.916 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:34.916 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:34.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.915+0000 7fd697b6c700 1 -- 172.21.15.195:0/1432586929 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd67c02ff60 msgr2=0x7fd67c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:34.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.915+0000 7fd697b6c700 1 --2- 172.21.15.195:0/1432586929 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd67c02ff60 0x7fd67c032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fd68c0096a0 tx=0x7fd68c006b40).stop 2023-10-12T22:39:34.918 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.915+0000 7fd697b6c700 1 -- 172.21.15.195:0/1432586929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6900f5410 msgr2=0x7fd690109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.915+0000 7fd697b6c700 1 --2- 172.21.15.195:0/1432586929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6900f5410 0x7fd690109450 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fd684013510 tx=0x7fd68400b040).stop 2023-10-12T22:39:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.915+0000 7fd697b6c700 1 -- 172.21.15.195:0/1432586929 shutdown_connections 2023-10-12T22:39:34.920 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.915+0000 7fd697b6c700 1 --2- 172.21.15.195:0/1432586929 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fd67c02ff60 0x7fd67c032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:34.920 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.915+0000 7fd697b6c700 1 --2- 172.21.15.195:0/1432586929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6900f5410 0x7fd690109450 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:34.921 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.915+0000 7fd697b6c700 1 -- 172.21.15.195:0/1432586929 >> 172.21.15.195:0/1432586929 conn(0x7fd6900f0b70 msgr2=0x7fd6900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:34.921 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.915+0000 7fd697b6c700 1 -- 172.21.15.195:0/1432586929 shutdown_connections 2023-10-12T22:39:34.921 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:34.915+0000 7fd697b6c700 1 -- 172.21.15.195:0/1432586929 wait complete. 2023-10-12T22:39:34.922 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:35.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:35 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:34.914953+0000 mon.smithi131 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.195:0/1432586929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:36.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:36.288 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:36.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:36 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:34.874427+0000 mgr.smithi131.efktma (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:37.426 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.423+0000 7f507759e700 1 -- 172.21.15.195:0/3120447037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50780f78e0 msgr2=0x7f50780f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:37.426 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.423+0000 7f507759e700 1 --2- 172.21.15.195:0/3120447037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50780f78e0 0x7f50780f7d00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f5068009fb0 tx=0x7f5068009b30).stop 2023-10-12T22:39:37.426 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.423+0000 7f507759e700 1 -- 172.21.15.195:0/3120447037 shutdown_connections 2023-10-12T22:39:37.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.423+0000 7f507759e700 1 --2- 172.21.15.195:0/3120447037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50780f78e0 0x7f50780f7d00 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:37.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.423+0000 7f507759e700 1 -- 172.21.15.195:0/3120447037 >> 172.21.15.195:0/3120447037 conn(0x7f50780f0120 msgr2=0x7f50780f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:37.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.423+0000 7f507759e700 1 -- 172.21.15.195:0/3120447037 shutdown_connections 2023-10-12T22:39:37.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.423+0000 7f507759e700 1 -- 172.21.15.195:0/3120447037 wait complete. 2023-10-12T22:39:37.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.423+0000 7f507759e700 1 Processor -- start 2023-10-12T22:39:37.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.423+0000 7f507759e700 1 -- start start 2023-10-12T22:39:37.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.423+0000 7f507759e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50780f78e0 0x7f50781064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:37.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.423+0000 7f507759e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50781069d0 con 0x7f50780f78e0 2023-10-12T22:39:37.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.424+0000 7f507659c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50780f78e0 0x7f50781064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:37.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.424+0000 7f507659c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50780f78e0 0x7f50781064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:56036/0 (socket says 172.21.15.195:56036) 2023-10-12T22:39:37.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.424+0000 7f507659c700 1 -- 172.21.15.195:0/443342115 learned_addr learned my addr 172.21.15.195:0/443342115 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:37.429 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.424+0000 7f507659c700 1 -- 172.21.15.195:0/443342115 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5068005040 con 0x7f50780f78e0 2023-10-12T22:39:37.429 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.424+0000 7f507659c700 1 --2- 172.21.15.195:0/443342115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50780f78e0 0x7f50781064e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f5068013510 tx=0x7f506800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:37.429 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.424+0000 7f506effd700 1 -- 172.21.15.195:0/443342115 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5068009320 con 0x7f50780f78e0 2023-10-12T22:39:37.429 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.424+0000 7f507759e700 1 -- 172.21.15.195:0/443342115 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5078106bd0 con 0x7f50780f78e0 2023-10-12T22:39:37.429 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.424+0000 7f506effd700 1 -- 172.21.15.195:0/443342115 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5068013f00 con 0x7f50780f78e0 2023-10-12T22:39:37.430 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.424+0000 7f506effd700 1 -- 172.21.15.195:0/443342115 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5068014dc0 con 0x7f50780f78e0 2023-10-12T22:39:37.430 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.424+0000 7f507759e700 1 -- 172.21.15.195:0/443342115 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5078107060 con 0x7f50780f78e0 2023-10-12T22:39:37.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.425+0000 7f506effd700 1 -- 172.21.15.195:0/443342115 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50680095f0 con 0x7f50780f78e0 2023-10-12T22:39:37.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.425+0000 7f507759e700 1 -- 172.21.15.195:0/443342115 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5078045ad0 con 0x7f50780f78e0 2023-10-12T22:39:37.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.425+0000 7f506effd700 1 --2- 172.21.15.195:0/443342115 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f506402ff60 0x7f5064032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:37.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.425+0000 7f506effd700 1 -- 172.21.15.195:0/443342115 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5068041000 con 0x7f50780f78e0 2023-10-12T22:39:37.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.426+0000 7f5075d9b700 1 --2- 172.21.15.195:0/443342115 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f506402ff60 0x7f5064032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:37.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.426+0000 7f5075d9b700 1 --2- 172.21.15.195:0/443342115 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f506402ff60 0x7f5064032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f50600096a0 tx=0x7f5060006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:37.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.430+0000 7f506effd700 1 -- 172.21.15.195:0/443342115 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5068014420 con 0x7f50780f78e0 2023-10-12T22:39:37.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.636+0000 7f507759e700 1 -- 172.21.15.195:0/443342115 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5078107f10 con 0x7f50780f78e0 2023-10-12T22:39:37.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.637+0000 7f506effd700 1 -- 172.21.15.195:0/443342115 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5068010230 con 0x7f50780f78e0 2023-10-12T22:39:37.638 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:37.638 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:37.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.638+0000 7f507759e700 1 -- 172.21.15.195:0/443342115 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f506402ff60 msgr2=0x7f5064032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:37.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.638+0000 7f507759e700 1 --2- 172.21.15.195:0/443342115 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f506402ff60 0x7f5064032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f50600096a0 tx=0x7f5060006b40).stop 2023-10-12T22:39:37.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.638+0000 7f507759e700 1 -- 172.21.15.195:0/443342115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50780f78e0 msgr2=0x7f50781064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:37.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.638+0000 7f507759e700 1 --2- 172.21.15.195:0/443342115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50780f78e0 0x7f50781064e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f5068013510 tx=0x7f506800b040).stop 2023-10-12T22:39:37.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.638+0000 7f507759e700 1 -- 172.21.15.195:0/443342115 shutdown_connections 2023-10-12T22:39:37.641 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.638+0000 7f507759e700 1 --2- 172.21.15.195:0/443342115 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f506402ff60 0x7f5064032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:37.641 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.638+0000 7f507759e700 1 --2- 172.21.15.195:0/443342115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50780f78e0 0x7f50781064e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:37.641 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.638+0000 7f507759e700 1 -- 172.21.15.195:0/443342115 >> 172.21.15.195:0/443342115 conn(0x7f50780f0120 msgr2=0x7f50780f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:37.641 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.638+0000 7f507759e700 1 -- 172.21.15.195:0/443342115 shutdown_connections 2023-10-12T22:39:37.641 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:37.639+0000 7f507759e700 1 -- 172.21.15.195:0/443342115 wait complete. 2023-10-12T22:39:37.642 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:38.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:38 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:36.874759+0000 mgr.smithi131.efktma (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:38.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:38 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:37.638397+0000 mon.smithi131 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.195:0/443342115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:39.024 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:39.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:40.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.226+0000 7f5141421700 1 -- 172.21.15.195:0/2267016737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f513c0f5410 msgr2=0x7f513c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:40.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.226+0000 7f5141421700 1 --2- 172.21.15.195:0/2267016737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f513c0f5410 0x7f513c0f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f5124009fb0 tx=0x7f5124009b30).stop 2023-10-12T22:39:40.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.226+0000 7f5141421700 1 -- 172.21.15.195:0/2267016737 shutdown_connections 2023-10-12T22:39:40.228 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.226+0000 7f5141421700 1 --2- 172.21.15.195:0/2267016737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f513c0f5410 0x7f513c0f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:40.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.226+0000 7f5141421700 1 -- 172.21.15.195:0/2267016737 >> 172.21.15.195:0/2267016737 conn(0x7f513c0f0b70 msgr2=0x7f513c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:40.229 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.226+0000 7f5141421700 1 -- 172.21.15.195:0/2267016737 shutdown_connections 2023-10-12T22:39:40.230 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.229+0000 7f5141421700 1 -- 172.21.15.195:0/2267016737 wait complete. 2023-10-12T22:39:40.231 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.230+0000 7f5141421700 1 Processor -- start 2023-10-12T22:39:40.231 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.230+0000 7f5141421700 1 -- start start 2023-10-12T22:39:40.231 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.230+0000 7f5141421700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f513c0f5410 0x7f513c106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:40.231 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.230+0000 7f5141421700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c106c50 con 0x7f513c0f5410 2023-10-12T22:39:40.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.230+0000 7f513affd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f513c0f5410 0x7f513c106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:40.233 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.230+0000 7f513affd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f513c0f5410 0x7f513c106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:54216/0 (socket says 172.21.15.195:54216) 2023-10-12T22:39:40.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.230+0000 7f513affd700 1 -- 172.21.15.195:0/1392295416 learned_addr learned my addr 172.21.15.195:0/1392295416 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:40.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.231+0000 7f513affd700 1 -- 172.21.15.195:0/1392295416 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5124005040 con 0x7f513c0f5410 2023-10-12T22:39:40.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.231+0000 7f513affd700 1 --2- 172.21.15.195:0/1392295416 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f513c0f5410 0x7f513c106760 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f5124009f80 tx=0x7f512400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:40.234 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.231+0000 7f51337fe700 1 -- 172.21.15.195:0/1392295416 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5124009320 con 0x7f513c0f5410 2023-10-12T22:39:40.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.231+0000 7f5141421700 1 -- 172.21.15.195:0/1392295416 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c106e50 con 0x7f513c0f5410 2023-10-12T22:39:40.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.231+0000 7f51337fe700 1 -- 172.21.15.195:0/1392295416 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5124013ec0 con 0x7f513c0f5410 2023-10-12T22:39:40.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.231+0000 7f51337fe700 1 -- 172.21.15.195:0/1392295416 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f512401ece0 con 0x7f513c0f5410 2023-10-12T22:39:40.235 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.231+0000 7f5141421700 1 -- 172.21.15.195:0/1392295416 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c1072e0 con 0x7f513c0f5410 2023-10-12T22:39:40.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.232+0000 7f51337fe700 1 -- 172.21.15.195:0/1392295416 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5124014420 con 0x7f513c0f5410 2023-10-12T22:39:40.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.232+0000 7f5141421700 1 -- 172.21.15.195:0/1392295416 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c045ad0 con 0x7f513c0f5410 2023-10-12T22:39:40.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.232+0000 7f51337fe700 1 --2- 172.21.15.195:0/1392295416 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f512802ffb0 0x7f5128032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:40.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.232+0000 7f51337fe700 1 -- 172.21.15.195:0/1392295416 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5124040fc0 con 0x7f513c0f5410 2023-10-12T22:39:40.236 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.232+0000 7f513a7fc700 1 --2- 172.21.15.195:0/1392295416 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f512802ffb0 0x7f5128032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:40.237 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.233+0000 7f513a7fc700 1 --2- 172.21.15.195:0/1392295416 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f512802ffb0 0x7f5128032470 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f512c0096a0 tx=0x7f512c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:40.238 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.237+0000 7f51337fe700 1 -- 172.21.15.195:0/1392295416 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f512406d2b0 con 0x7f513c0f5410 2023-10-12T22:39:40.441 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.440+0000 7f5141421700 1 -- 172.21.15.195:0/1392295416 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f513c108190 con 0x7f513c0f5410 2023-10-12T22:39:40.442 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.440+0000 7f51337fe700 1 -- 172.21.15.195:0/1392295416 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f512402c2d0 con 0x7f513c0f5410 2023-10-12T22:39:40.442 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:40.442 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:40.443 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.442+0000 7f5141421700 1 -- 172.21.15.195:0/1392295416 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f512802ffb0 msgr2=0x7f5128032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:40.443 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.442+0000 7f5141421700 1 --2- 172.21.15.195:0/1392295416 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f512802ffb0 0x7f5128032470 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f512c0096a0 tx=0x7f512c006b40).stop 2023-10-12T22:39:40.443 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.442+0000 7f5141421700 1 -- 172.21.15.195:0/1392295416 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f513c0f5410 msgr2=0x7f513c106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:40.444 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.442+0000 7f5141421700 1 --2- 172.21.15.195:0/1392295416 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f513c0f5410 0x7f513c106760 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f5124009f80 tx=0x7f512400b040).stop 2023-10-12T22:39:40.444 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.442+0000 7f5141421700 1 -- 172.21.15.195:0/1392295416 shutdown_connections 2023-10-12T22:39:40.444 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.442+0000 7f5141421700 1 --2- 172.21.15.195:0/1392295416 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f512802ffb0 0x7f5128032470 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:40.444 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.442+0000 7f5141421700 1 --2- 172.21.15.195:0/1392295416 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f513c0f5410 0x7f513c106760 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:40.444 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.442+0000 7f5141421700 1 -- 172.21.15.195:0/1392295416 >> 172.21.15.195:0/1392295416 conn(0x7f513c0f0b70 msgr2=0x7f513c05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:40.445 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.442+0000 7f5141421700 1 -- 172.21.15.195:0/1392295416 shutdown_connections 2023-10-12T22:39:40.445 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:40.442+0000 7f5141421700 1 -- 172.21.15.195:0/1392295416 wait complete. 2023-10-12T22:39:40.445 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:40.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:40 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:38.875076+0000 mgr.smithi131.efktma (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:41.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:41 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:40.441849+0000 mon.smithi131 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.195:0/1392295416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:41.824 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:41.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:42.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:42 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:40.875402+0000 mgr.smithi131.efktma (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:42.981 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.979+0000 7f60ae04b700 1 -- 172.21.15.195:0/923927191 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60a80f5410 msgr2=0x7f60a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:42.982 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.979+0000 7f60ae04b700 1 --2- 172.21.15.195:0/923927191 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60a80f5410 0x7f60a80f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f6098009fb0 tx=0x7f6098009b30).stop 2023-10-12T22:39:42.982 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.979+0000 7f60ae04b700 1 -- 172.21.15.195:0/923927191 shutdown_connections 2023-10-12T22:39:42.982 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.979+0000 7f60ae04b700 1 --2- 172.21.15.195:0/923927191 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60a80f5410 0x7f60a80f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:42.982 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.979+0000 7f60ae04b700 1 -- 172.21.15.195:0/923927191 >> 172.21.15.195:0/923927191 conn(0x7f60a80f0b70 msgr2=0x7f60a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:42.983 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.979+0000 7f60ae04b700 1 -- 172.21.15.195:0/923927191 shutdown_connections 2023-10-12T22:39:42.983 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.979+0000 7f60ae04b700 1 -- 172.21.15.195:0/923927191 wait complete. 2023-10-12T22:39:42.983 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.980+0000 7f60ae04b700 1 Processor -- start 2023-10-12T22:39:42.983 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.980+0000 7f60ae04b700 1 -- start start 2023-10-12T22:39:42.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.981+0000 7f60ae04b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60a80f5410 0x7f60a8108930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:42.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.981+0000 7f60ae04b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a8108e20 con 0x7f60a80f5410 2023-10-12T22:39:42.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.981+0000 7f60a77fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60a80f5410 0x7f60a8108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:42.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.981+0000 7f60a77fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60a80f5410 0x7f60a8108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:54242/0 (socket says 172.21.15.195:54242) 2023-10-12T22:39:42.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.981+0000 7f60a77fe700 1 -- 172.21.15.195:0/1118310500 learned_addr learned my addr 172.21.15.195:0/1118310500 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:42.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.981+0000 7f60a77fe700 1 -- 172.21.15.195:0/1118310500 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6098005040 con 0x7f60a80f5410 2023-10-12T22:39:42.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.981+0000 7f60a77fe700 1 --2- 172.21.15.195:0/1118310500 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60a80f5410 0x7f60a8108930 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f6098013510 tx=0x7f609800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:42.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.981+0000 7f608ffff700 1 -- 172.21.15.195:0/1118310500 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6098009320 con 0x7f60a80f5410 2023-10-12T22:39:42.987 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.981+0000 7f60ae04b700 1 -- 172.21.15.195:0/1118310500 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a8109020 con 0x7f60a80f5410 2023-10-12T22:39:42.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.981+0000 7f608ffff700 1 -- 172.21.15.195:0/1118310500 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6098013f00 con 0x7f60a80f5410 2023-10-12T22:39:42.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.981+0000 7f608ffff700 1 -- 172.21.15.195:0/1118310500 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6098014d90 con 0x7f60a80f5410 2023-10-12T22:39:42.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.981+0000 7f60ae04b700 1 -- 172.21.15.195:0/1118310500 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a81094b0 con 0x7f60a80f5410 2023-10-12T22:39:42.988 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.982+0000 7f608ffff700 1 -- 172.21.15.195:0/1118310500 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60980263a0 con 0x7f60a80f5410 2023-10-12T22:39:42.989 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.982+0000 7f60ae04b700 1 -- 172.21.15.195:0/1118310500 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a8045ad0 con 0x7f60a80f5410 2023-10-12T22:39:42.989 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.983+0000 7f608ffff700 1 --2- 172.21.15.195:0/1118310500 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f609002ffb0 0x7f6090032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:42.989 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.983+0000 7f608ffff700 1 -- 172.21.15.195:0/1118310500 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60980412b0 con 0x7f60a80f5410 2023-10-12T22:39:42.989 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.983+0000 7f60a6ffd700 1 --2- 172.21.15.195:0/1118310500 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f609002ffb0 0x7f6090032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:42.990 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.983+0000 7f60a6ffd700 1 --2- 172.21.15.195:0/1118310500 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f609002ffb0 0x7f6090032470 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f609c0096a0 tx=0x7f609c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:42.990 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:42.987+0000 7f608ffff700 1 -- 172.21.15.195:0/1118310500 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f609801d380 con 0x7f60a80f5410 2023-10-12T22:39:43.198 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:43.197+0000 7f60ae04b700 1 -- 172.21.15.195:0/1118310500 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60a80fe190 con 0x7f60a80f5410 2023-10-12T22:39:43.199 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:43.198+0000 7f608ffff700 1 -- 172.21.15.195:0/1118310500 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f609801cb10 con 0x7f60a80f5410 2023-10-12T22:39:43.199 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:43.200 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:43.201 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:43.199+0000 7f60ae04b700 1 -- 172.21.15.195:0/1118310500 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f609002ffb0 msgr2=0x7f6090032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:43.201 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:43.199+0000 7f60ae04b700 1 --2- 172.21.15.195:0/1118310500 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f609002ffb0 0x7f6090032470 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f609c0096a0 tx=0x7f609c006b40).stop 2023-10-12T22:39:43.201 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:43.199+0000 7f60ae04b700 1 -- 172.21.15.195:0/1118310500 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60a80f5410 msgr2=0x7f60a8108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:43.201 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:43.199+0000 7f60ae04b700 1 --2- 172.21.15.195:0/1118310500 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60a80f5410 0x7f60a8108930 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f6098013510 tx=0x7f609800b040).stop 2023-10-12T22:39:43.202 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:43.199+0000 7f60ae04b700 1 -- 172.21.15.195:0/1118310500 shutdown_connections 2023-10-12T22:39:43.202 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:43.200+0000 7f60ae04b700 1 --2- 172.21.15.195:0/1118310500 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f609002ffb0 0x7f6090032470 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:43.202 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:43.200+0000 7f60ae04b700 1 --2- 172.21.15.195:0/1118310500 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60a80f5410 0x7f60a8108930 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:43.202 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:43.200+0000 7f60ae04b700 1 -- 172.21.15.195:0/1118310500 >> 172.21.15.195:0/1118310500 conn(0x7f60a80f0b70 msgr2=0x7f60a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:43.202 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:43.200+0000 7f60ae04b700 1 -- 172.21.15.195:0/1118310500 shutdown_connections 2023-10-12T22:39:43.203 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:43.200+0000 7f60ae04b700 1 -- 172.21.15.195:0/1118310500 wait complete. 2023-10-12T22:39:43.203 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:44.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:44 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:42.875644+0000 mgr.smithi131.efktma (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:44.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:44 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:43.199269+0000 mon.smithi131 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.195:0/1118310500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:44.594 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:44.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:45.729 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.728+0000 7ffb16271700 1 -- 172.21.15.195:0/113114057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb100f62a0 msgr2=0x7ffb100f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:45.729 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.728+0000 7ffb16271700 1 --2- 172.21.15.195:0/113114057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb100f62a0 0x7ffb100f66c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7ffb0c009fb0 tx=0x7ffb0c009b30).stop 2023-10-12T22:39:45.730 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.728+0000 7ffb16271700 1 -- 172.21.15.195:0/113114057 shutdown_connections 2023-10-12T22:39:45.730 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.728+0000 7ffb16271700 1 --2- 172.21.15.195:0/113114057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb100f62a0 0x7ffb100f66c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:45.730 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.728+0000 7ffb16271700 1 -- 172.21.15.195:0/113114057 >> 172.21.15.195:0/113114057 conn(0x7ffb100f1aa0 msgr2=0x7ffb100f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:45.730 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.728+0000 7ffb16271700 1 -- 172.21.15.195:0/113114057 shutdown_connections 2023-10-12T22:39:45.730 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.728+0000 7ffb16271700 1 -- 172.21.15.195:0/113114057 wait complete. 2023-10-12T22:39:45.732 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.728+0000 7ffb16271700 1 Processor -- start 2023-10-12T22:39:45.733 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.728+0000 7ffb16271700 1 -- start start 2023-10-12T22:39:45.733 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.729+0000 7ffb16271700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb100f62a0 0x7ffb101064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:45.733 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.729+0000 7ffb16271700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb101069a0 con 0x7ffb100f62a0 2023-10-12T22:39:45.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.729+0000 7ffb1526f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb100f62a0 0x7ffb101064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:45.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.729+0000 7ffb1526f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb100f62a0 0x7ffb101064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:54260/0 (socket says 172.21.15.195:54260) 2023-10-12T22:39:45.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.729+0000 7ffb1526f700 1 -- 172.21.15.195:0/3712284185 learned_addr learned my addr 172.21.15.195:0/3712284185 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:45.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.729+0000 7ffb1526f700 1 -- 172.21.15.195:0/3712284185 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c005040 con 0x7ffb100f62a0 2023-10-12T22:39:45.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.729+0000 7ffb1526f700 1 --2- 172.21.15.195:0/3712284185 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb100f62a0 0x7ffb101064b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7ffb0c013510 tx=0x7ffb0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:45.735 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.730+0000 7ffb05ffb700 1 -- 172.21.15.195:0/3712284185 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb0c009320 con 0x7ffb100f62a0 2023-10-12T22:39:45.735 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.730+0000 7ffb16271700 1 -- 172.21.15.195:0/3712284185 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb10106ba0 con 0x7ffb100f62a0 2023-10-12T22:39:45.735 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.730+0000 7ffb05ffb700 1 -- 172.21.15.195:0/3712284185 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb0c013f00 con 0x7ffb100f62a0 2023-10-12T22:39:45.736 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.730+0000 7ffb05ffb700 1 -- 172.21.15.195:0/3712284185 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb0c014dc0 con 0x7ffb100f62a0 2023-10-12T22:39:45.736 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.730+0000 7ffb16271700 1 -- 172.21.15.195:0/3712284185 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb10107030 con 0x7ffb100f62a0 2023-10-12T22:39:45.736 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.731+0000 7ffb05ffb700 1 -- 172.21.15.195:0/3712284185 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb0c01d3d0 con 0x7ffb100f62a0 2023-10-12T22:39:45.736 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.731+0000 7ffb05ffb700 1 --2- 172.21.15.195:0/3712284185 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffafc02ff60 0x7ffafc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:45.736 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.731+0000 7ffb05ffb700 1 -- 172.21.15.195:0/3712284185 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb0c041010 con 0x7ffb100f62a0 2023-10-12T22:39:45.737 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.731+0000 7ffb16271700 1 -- 172.21.15.195:0/3712284185 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb10045ad0 con 0x7ffb100f62a0 2023-10-12T22:39:45.737 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.731+0000 7ffb14a6e700 1 --2- 172.21.15.195:0/3712284185 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffafc02ff60 0x7ffafc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:45.737 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.731+0000 7ffb14a6e700 1 --2- 172.21.15.195:0/3712284185 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffafc02ff60 0x7ffafc032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ffb000096a0 tx=0x7ffb00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:45.738 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.736+0000 7ffb05ffb700 1 -- 172.21.15.195:0/3712284185 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb0c012920 con 0x7ffb100f62a0 2023-10-12T22:39:45.947 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.946+0000 7ffb16271700 1 -- 172.21.15.195:0/3712284185 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb10107e80 con 0x7ffb100f62a0 2023-10-12T22:39:45.948 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.947+0000 7ffb05ffb700 1 -- 172.21.15.195:0/3712284185 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffb0c02b020 con 0x7ffb100f62a0 2023-10-12T22:39:45.949 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:45.949 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:45.950 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.948+0000 7ffb16271700 1 -- 172.21.15.195:0/3712284185 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffafc02ff60 msgr2=0x7ffafc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:45.950 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.948+0000 7ffb16271700 1 --2- 172.21.15.195:0/3712284185 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffafc02ff60 0x7ffafc032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ffb000096a0 tx=0x7ffb00006b40).stop 2023-10-12T22:39:45.950 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.948+0000 7ffb16271700 1 -- 172.21.15.195:0/3712284185 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb100f62a0 msgr2=0x7ffb101064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:45.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.948+0000 7ffb16271700 1 --2- 172.21.15.195:0/3712284185 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb100f62a0 0x7ffb101064b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7ffb0c013510 tx=0x7ffb0c00b040).stop 2023-10-12T22:39:45.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.948+0000 7ffb16271700 1 -- 172.21.15.195:0/3712284185 shutdown_connections 2023-10-12T22:39:45.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.948+0000 7ffb16271700 1 --2- 172.21.15.195:0/3712284185 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffafc02ff60 0x7ffafc032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:45.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.948+0000 7ffb16271700 1 --2- 172.21.15.195:0/3712284185 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb100f62a0 0x7ffb101064b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:45.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.948+0000 7ffb16271700 1 -- 172.21.15.195:0/3712284185 >> 172.21.15.195:0/3712284185 conn(0x7ffb100f1aa0 msgr2=0x7ffb100f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:45.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.948+0000 7ffb16271700 1 -- 172.21.15.195:0/3712284185 shutdown_connections 2023-10-12T22:39:45.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:45.948+0000 7ffb16271700 1 -- 172.21.15.195:0/3712284185 wait complete. 2023-10-12T22:39:45.952 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:46.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:46 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:44.875990+0000 mgr.smithi131.efktma (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:46.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:46 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:45.948280+0000 mon.smithi131 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.195:0/3712284185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:47.316 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:47.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:48.469 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.467+0000 7fdc5b10a700 1 -- 172.21.15.195:0/1793890820 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc540f6510 msgr2=0x7fdc540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.467+0000 7fdc5b10a700 1 --2- 172.21.15.195:0/1793890820 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc540f6510 0x7fdc540f6930 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fdc48009fb0 tx=0x7fdc48009b30).stop 2023-10-12T22:39:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.467+0000 7fdc5b10a700 1 -- 172.21.15.195:0/1793890820 shutdown_connections 2023-10-12T22:39:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.467+0000 7fdc5b10a700 1 --2- 172.21.15.195:0/1793890820 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc540f6510 0x7fdc540f6930 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.467+0000 7fdc5b10a700 1 -- 172.21.15.195:0/1793890820 >> 172.21.15.195:0/1793890820 conn(0x7fdc540f1c70 msgr2=0x7fdc540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.467+0000 7fdc5b10a700 1 -- 172.21.15.195:0/1793890820 shutdown_connections 2023-10-12T22:39:48.471 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.467+0000 7fdc5b10a700 1 -- 172.21.15.195:0/1793890820 wait complete. 2023-10-12T22:39:48.471 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.468+0000 7fdc5b10a700 1 Processor -- start 2023-10-12T22:39:48.471 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.468+0000 7fdc5b10a700 1 -- start start 2023-10-12T22:39:48.471 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.468+0000 7fdc5b10a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc540f6510 0x7fdc5406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:48.471 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.468+0000 7fdc5b10a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc54069e20 con 0x7fdc540f6510 2023-10-12T22:39:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.469+0000 7fdc58ea6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc540f6510 0x7fdc5406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.469+0000 7fdc58ea6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc540f6510 0x7fdc5406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:60704/0 (socket says 172.21.15.195:60704) 2023-10-12T22:39:48.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.469+0000 7fdc58ea6700 1 -- 172.21.15.195:0/3210875509 learned_addr learned my addr 172.21.15.195:0/3210875509 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:48.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.469+0000 7fdc58ea6700 1 -- 172.21.15.195:0/3210875509 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc48005040 con 0x7fdc540f6510 2023-10-12T22:39:48.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.469+0000 7fdc58ea6700 1 --2- 172.21.15.195:0/3210875509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc540f6510 0x7fdc5406b9a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fdc48013510 tx=0x7fdc4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:48.474 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.469+0000 7fdc517fa700 1 -- 172.21.15.195:0/3210875509 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc48009320 con 0x7fdc540f6510 2023-10-12T22:39:48.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.469+0000 7fdc517fa700 1 -- 172.21.15.195:0/3210875509 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc48013f00 con 0x7fdc540f6510 2023-10-12T22:39:48.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.469+0000 7fdc5b10a700 1 -- 172.21.15.195:0/3210875509 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc54069fc0 con 0x7fdc540f6510 2023-10-12T22:39:48.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.469+0000 7fdc517fa700 1 -- 172.21.15.195:0/3210875509 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc48014d90 con 0x7fdc540f6510 2023-10-12T22:39:48.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.469+0000 7fdc5b10a700 1 -- 172.21.15.195:0/3210875509 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5406a450 con 0x7fdc540f6510 2023-10-12T22:39:48.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.470+0000 7fdc517fa700 1 -- 172.21.15.195:0/3210875509 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc480095f0 con 0x7fdc540f6510 2023-10-12T22:39:48.475 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.470+0000 7fdc5b10a700 1 -- 172.21.15.195:0/3210875509 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc54059100 con 0x7fdc540f6510 2023-10-12T22:39:48.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.470+0000 7fdc517fa700 1 --2- 172.21.15.195:0/3210875509 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdc4402ff60 0x7fdc44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:48.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.470+0000 7fdc517fa700 1 -- 172.21.15.195:0/3210875509 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc480410a0 con 0x7fdc540f6510 2023-10-12T22:39:48.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.471+0000 7fdc53fff700 1 --2- 172.21.15.195:0/3210875509 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdc4402ff60 0x7fdc44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:48.476 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.471+0000 7fdc53fff700 1 --2- 172.21.15.195:0/3210875509 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdc4402ff60 0x7fdc44032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fdc400096a0 tx=0x7fdc40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:48.477 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.475+0000 7fdc517fa700 1 -- 172.21.15.195:0/3210875509 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc4801d7f0 con 0x7fdc540f6510 2023-10-12T22:39:48.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:48 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:46.876378+0000 mgr.smithi131.efktma (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.683+0000 7fdc5b10a700 1 -- 172.21.15.195:0/3210875509 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc540fbf60 con 0x7fdc540f6510 2023-10-12T22:39:48.685 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.683+0000 7fdc517fa700 1 -- 172.21.15.195:0/3210875509 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdc48010230 con 0x7fdc540f6510 2023-10-12T22:39:48.685 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:48.685 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:48.686 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.685+0000 7fdc5b10a700 1 -- 172.21.15.195:0/3210875509 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdc4402ff60 msgr2=0x7fdc44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:48.687 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.685+0000 7fdc5b10a700 1 --2- 172.21.15.195:0/3210875509 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdc4402ff60 0x7fdc44032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fdc400096a0 tx=0x7fdc40006b40).stop 2023-10-12T22:39:48.687 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.685+0000 7fdc5b10a700 1 -- 172.21.15.195:0/3210875509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc540f6510 msgr2=0x7fdc5406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:48.687 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.685+0000 7fdc5b10a700 1 --2- 172.21.15.195:0/3210875509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc540f6510 0x7fdc5406b9a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fdc48013510 tx=0x7fdc4800b040).stop 2023-10-12T22:39:48.687 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.685+0000 7fdc5b10a700 1 -- 172.21.15.195:0/3210875509 shutdown_connections 2023-10-12T22:39:48.687 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.685+0000 7fdc5b10a700 1 --2- 172.21.15.195:0/3210875509 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fdc4402ff60 0x7fdc44032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:48.688 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.685+0000 7fdc5b10a700 1 --2- 172.21.15.195:0/3210875509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc540f6510 0x7fdc5406b9a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:48.688 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.685+0000 7fdc5b10a700 1 -- 172.21.15.195:0/3210875509 >> 172.21.15.195:0/3210875509 conn(0x7fdc540f1c70 msgr2=0x7fdc540f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:48.688 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.685+0000 7fdc5b10a700 1 -- 172.21.15.195:0/3210875509 shutdown_connections 2023-10-12T22:39:48.688 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:48.685+0000 7fdc5b10a700 1 -- 172.21.15.195:0/3210875509 wait complete. 2023-10-12T22:39:48.688 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:49.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:49 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:48.684761+0000 mon.smithi131 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.195:0/3210875509' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:50.101 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:50.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:50.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:50 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:48.876801+0000 mgr.smithi131.efktma (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:51.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.267+0000 7ffb24165700 1 -- 172.21.15.195:0/1500605586 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb1c0f5410 msgr2=0x7ffb1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:51.269 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.267+0000 7ffb24165700 1 --2- 172.21.15.195:0/1500605586 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb1c0f5410 0x7ffb1c0f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7ffb0c009fb0 tx=0x7ffb0c009b30).stop 2023-10-12T22:39:51.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.267+0000 7ffb24165700 1 -- 172.21.15.195:0/1500605586 shutdown_connections 2023-10-12T22:39:51.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.267+0000 7ffb24165700 1 --2- 172.21.15.195:0/1500605586 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb1c0f5410 0x7ffb1c0f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:51.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.267+0000 7ffb24165700 1 -- 172.21.15.195:0/1500605586 >> 172.21.15.195:0/1500605586 conn(0x7ffb1c0f0b70 msgr2=0x7ffb1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:51.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.267+0000 7ffb24165700 1 -- 172.21.15.195:0/1500605586 shutdown_connections 2023-10-12T22:39:51.270 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.267+0000 7ffb24165700 1 -- 172.21.15.195:0/1500605586 wait complete. 2023-10-12T22:39:51.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.267+0000 7ffb24165700 1 Processor -- start 2023-10-12T22:39:51.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.267+0000 7ffb24165700 1 -- start start 2023-10-12T22:39:51.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.267+0000 7ffb24165700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb1c0f5410 0x7ffb1c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:51.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.267+0000 7ffb24165700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c106c10 con 0x7ffb1c0f5410 2023-10-12T22:39:51.271 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.268+0000 7ffb21f01700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb1c0f5410 0x7ffb1c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:51.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.268+0000 7ffb21f01700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb1c0f5410 0x7ffb1c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:60720/0 (socket says 172.21.15.195:60720) 2023-10-12T22:39:51.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.268+0000 7ffb21f01700 1 -- 172.21.15.195:0/2116426963 learned_addr learned my addr 172.21.15.195:0/2116426963 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:51.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.268+0000 7ffb21f01700 1 -- 172.21.15.195:0/2116426963 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c005040 con 0x7ffb1c0f5410 2023-10-12T22:39:51.272 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.268+0000 7ffb21f01700 1 --2- 172.21.15.195:0/2116426963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb1c0f5410 0x7ffb1c106720 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7ffb0c013510 tx=0x7ffb0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:51.273 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.268+0000 7ffb127fc700 1 -- 172.21.15.195:0/2116426963 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb0c009320 con 0x7ffb1c0f5410 2023-10-12T22:39:51.274 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.268+0000 7ffb24165700 1 -- 172.21.15.195:0/2116426963 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c106e10 con 0x7ffb1c0f5410 2023-10-12T22:39:51.274 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.268+0000 7ffb127fc700 1 -- 172.21.15.195:0/2116426963 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb0c013f00 con 0x7ffb1c0f5410 2023-10-12T22:39:51.274 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.268+0000 7ffb24165700 1 -- 172.21.15.195:0/2116426963 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c1072a0 con 0x7ffb1c0f5410 2023-10-12T22:39:51.274 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.268+0000 7ffb127fc700 1 -- 172.21.15.195:0/2116426963 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb0c014d90 con 0x7ffb1c0f5410 2023-10-12T22:39:51.274 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.269+0000 7ffb127fc700 1 -- 172.21.15.195:0/2116426963 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb0c0095f0 con 0x7ffb1c0f5410 2023-10-12T22:39:51.275 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.269+0000 7ffb24165700 1 -- 172.21.15.195:0/2116426963 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c045ad0 con 0x7ffb1c0f5410 2023-10-12T22:39:51.275 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.269+0000 7ffb127fc700 1 --2- 172.21.15.195:0/2116426963 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffb0802ff60 0x7ffb08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:51.275 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.269+0000 7ffb127fc700 1 -- 172.21.15.195:0/2116426963 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb0c041090 con 0x7ffb1c0f5410 2023-10-12T22:39:51.275 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.270+0000 7ffb21700700 1 --2- 172.21.15.195:0/2116426963 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffb0802ff60 0x7ffb08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:51.275 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.270+0000 7ffb21700700 1 --2- 172.21.15.195:0/2116426963 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffb0802ff60 0x7ffb08032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7ffb180096a0 tx=0x7ffb18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:51.276 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.275+0000 7ffb127fc700 1 -- 172.21.15.195:0/2116426963 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb0c014420 con 0x7ffb1c0f5410 2023-10-12T22:39:51.484 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.483+0000 7ffb24165700 1 -- 172.21.15.195:0/2116426963 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb1c1080f0 con 0x7ffb1c0f5410 2023-10-12T22:39:51.485 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.484+0000 7ffb127fc700 1 -- 172.21.15.195:0/2116426963 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffb0c010230 con 0x7ffb1c0f5410 2023-10-12T22:39:51.486 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:51.486 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:51.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.485+0000 7ffb24165700 1 -- 172.21.15.195:0/2116426963 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffb0802ff60 msgr2=0x7ffb08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:51.487 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.485+0000 7ffb24165700 1 --2- 172.21.15.195:0/2116426963 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffb0802ff60 0x7ffb08032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7ffb180096a0 tx=0x7ffb18006b40).stop 2023-10-12T22:39:51.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.485+0000 7ffb24165700 1 -- 172.21.15.195:0/2116426963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb1c0f5410 msgr2=0x7ffb1c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:51.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.485+0000 7ffb24165700 1 --2- 172.21.15.195:0/2116426963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb1c0f5410 0x7ffb1c106720 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7ffb0c013510 tx=0x7ffb0c00b040).stop 2023-10-12T22:39:51.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.485+0000 7ffb24165700 1 -- 172.21.15.195:0/2116426963 shutdown_connections 2023-10-12T22:39:51.488 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.485+0000 7ffb24165700 1 --2- 172.21.15.195:0/2116426963 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7ffb0802ff60 0x7ffb08032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:51.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.485+0000 7ffb24165700 1 --2- 172.21.15.195:0/2116426963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb1c0f5410 0x7ffb1c106720 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:51.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.485+0000 7ffb24165700 1 -- 172.21.15.195:0/2116426963 >> 172.21.15.195:0/2116426963 conn(0x7ffb1c0f0b70 msgr2=0x7ffb1c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:51.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.485+0000 7ffb24165700 1 -- 172.21.15.195:0/2116426963 shutdown_connections 2023-10-12T22:39:51.489 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:51.485+0000 7ffb24165700 1 -- 172.21.15.195:0/2116426963 wait complete. 2023-10-12T22:39:51.489 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:52.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:50.877121+0000 mgr.smithi131.efktma (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:52.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:52 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:51.484981+0000 mon.smithi131 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.195:0/2116426963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:52.860 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:52.861 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:54.104 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.102+0000 7f99e0c07700 1 -- 172.21.15.195:0/3936130237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99dc060200 msgr2=0x7f99dc0f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:54.104 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.102+0000 7f99e0c07700 1 --2- 172.21.15.195:0/3936130237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99dc060200 0x7f99dc0f9170 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f99cc009fb0 tx=0x7f99cc009b30).stop 2023-10-12T22:39:54.104 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.102+0000 7f99e0c07700 1 -- 172.21.15.195:0/3936130237 shutdown_connections 2023-10-12T22:39:54.105 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.102+0000 7f99e0c07700 1 --2- 172.21.15.195:0/3936130237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99dc060200 0x7f99dc0f9170 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:54.105 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.102+0000 7f99e0c07700 1 -- 172.21.15.195:0/3936130237 >> 172.21.15.195:0/3936130237 conn(0x7f99dc0f0b70 msgr2=0x7f99dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:54.105 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.102+0000 7f99e0c07700 1 -- 172.21.15.195:0/3936130237 shutdown_connections 2023-10-12T22:39:54.105 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.102+0000 7f99e0c07700 1 -- 172.21.15.195:0/3936130237 wait complete. 2023-10-12T22:39:54.105 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.103+0000 7f99e0c07700 1 Processor -- start 2023-10-12T22:39:54.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.103+0000 7f99e0c07700 1 -- start start 2023-10-12T22:39:54.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.103+0000 7f99e0c07700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99dc060200 0x7f99dc0f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:54.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.103+0000 7f99e0c07700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc0f5070 con 0x7f99dc060200 2023-10-12T22:39:54.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.103+0000 7f99da59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99dc060200 0x7f99dc0f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:54.106 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.103+0000 7f99da59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99dc060200 0x7f99dc0f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:60738/0 (socket says 172.21.15.195:60738) 2023-10-12T22:39:54.107 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.103+0000 7f99da59c700 1 -- 172.21.15.195:0/3573906877 learned_addr learned my addr 172.21.15.195:0/3573906877 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:54.107 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.103+0000 7f99da59c700 1 -- 172.21.15.195:0/3573906877 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99cc005040 con 0x7f99dc060200 2023-10-12T22:39:54.107 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.103+0000 7f99da59c700 1 --2- 172.21.15.195:0/3573906877 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99dc060200 0x7f99dc0f6bf0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f99cc013510 tx=0x7f99cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:54.107 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.104+0000 7f99caffd700 1 -- 172.21.15.195:0/3573906877 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99cc009320 con 0x7f99dc060200 2023-10-12T22:39:54.107 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.104+0000 7f99e0c07700 1 -- 172.21.15.195:0/3573906877 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99dc0f5210 con 0x7f99dc060200 2023-10-12T22:39:54.107 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.104+0000 7f99caffd700 1 -- 172.21.15.195:0/3573906877 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99cc013f00 con 0x7f99dc060200 2023-10-12T22:39:54.108 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.104+0000 7f99caffd700 1 -- 172.21.15.195:0/3573906877 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99cc014dc0 con 0x7f99dc060200 2023-10-12T22:39:54.108 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.104+0000 7f99e0c07700 1 -- 172.21.15.195:0/3573906877 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99dc0f56a0 con 0x7f99dc060200 2023-10-12T22:39:54.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.104+0000 7f99caffd700 1 -- 172.21.15.195:0/3573906877 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99cc0095f0 con 0x7f99dc060200 2023-10-12T22:39:54.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.104+0000 7f99e0c07700 1 -- 172.21.15.195:0/3573906877 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99dc059100 con 0x7f99dc060200 2023-10-12T22:39:54.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.105+0000 7f99caffd700 1 --2- 172.21.15.195:0/3573906877 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f99c402ff60 0x7f99c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:54.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.105+0000 7f99caffd700 1 -- 172.21.15.195:0/3573906877 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f99cc041040 con 0x7f99dc060200 2023-10-12T22:39:54.110 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.105+0000 7f99d9d9b700 1 --2- 172.21.15.195:0/3573906877 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f99c402ff60 0x7f99c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:54.110 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.105+0000 7f99d9d9b700 1 --2- 172.21.15.195:0/3573906877 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f99c402ff60 0x7f99c4032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f99d00096a0 tx=0x7f99d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:54.110 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.109+0000 7f99caffd700 1 -- 172.21.15.195:0/3573906877 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99cc014420 con 0x7f99dc060200 2023-10-12T22:39:54.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.314+0000 7f99e0c07700 1 -- 172.21.15.195:0/3573906877 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99dc0fbf50 con 0x7f99dc060200 2023-10-12T22:39:54.314 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.314+0000 7f99caffd700 1 -- 172.21.15.195:0/3573906877 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f99cc010230 con 0x7f99dc060200 2023-10-12T22:39:54.315 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:54.315 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:54.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.316+0000 7f99e0c07700 1 -- 172.21.15.195:0/3573906877 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f99c402ff60 msgr2=0x7f99c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:54.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.316+0000 7f99e0c07700 1 --2- 172.21.15.195:0/3573906877 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f99c402ff60 0x7f99c4032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f99d00096a0 tx=0x7f99d0006b40).stop 2023-10-12T22:39:54.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.316+0000 7f99e0c07700 1 -- 172.21.15.195:0/3573906877 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99dc060200 msgr2=0x7f99dc0f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:54.316 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.316+0000 7f99e0c07700 1 --2- 172.21.15.195:0/3573906877 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99dc060200 0x7f99dc0f6bf0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f99cc013510 tx=0x7f99cc00b040).stop 2023-10-12T22:39:54.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.316+0000 7f99e0c07700 1 -- 172.21.15.195:0/3573906877 shutdown_connections 2023-10-12T22:39:54.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.316+0000 7f99e0c07700 1 --2- 172.21.15.195:0/3573906877 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f99c402ff60 0x7f99c4032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:54.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.316+0000 7f99e0c07700 1 --2- 172.21.15.195:0/3573906877 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99dc060200 0x7f99dc0f6bf0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:54.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.316+0000 7f99e0c07700 1 -- 172.21.15.195:0/3573906877 >> 172.21.15.195:0/3573906877 conn(0x7f99dc0f0b70 msgr2=0x7f99dc0f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:54.317 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.316+0000 7f99e0c07700 1 -- 172.21.15.195:0/3573906877 shutdown_connections 2023-10-12T22:39:54.318 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:54.316+0000 7f99e0c07700 1 -- 172.21.15.195:0/3573906877 wait complete. 2023-10-12T22:39:54.318 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:54.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:54 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:52.877401+0000 mgr.smithi131.efktma (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:55.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:55 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:54.314633+0000 mon.smithi131 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.195:0/3573906877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:55.700 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:55.700 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:56.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:56 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:54.877760+0000 mgr.smithi131.efktma (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:56.910 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.908+0000 7fcce0280700 1 -- 172.21.15.195:0/3536542565 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd8060280 msgr2=0x7fccd80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:56.910 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.908+0000 7fcce0280700 1 --2- 172.21.15.195:0/3536542565 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd8060280 0x7fccd80f90f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fccd4009fb0 tx=0x7fccd4009b30).stop 2023-10-12T22:39:56.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.908+0000 7fcce0280700 1 -- 172.21.15.195:0/3536542565 shutdown_connections 2023-10-12T22:39:56.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.908+0000 7fcce0280700 1 --2- 172.21.15.195:0/3536542565 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd8060280 0x7fccd80f90f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:56.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.908+0000 7fcce0280700 1 -- 172.21.15.195:0/3536542565 >> 172.21.15.195:0/3536542565 conn(0x7fccd80f0b70 msgr2=0x7fccd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:56.911 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.908+0000 7fcce0280700 1 -- 172.21.15.195:0/3536542565 shutdown_connections 2023-10-12T22:39:56.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.908+0000 7fcce0280700 1 -- 172.21.15.195:0/3536542565 wait complete. 2023-10-12T22:39:56.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.908+0000 7fcce0280700 1 Processor -- start 2023-10-12T22:39:56.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.909+0000 7fcce0280700 1 -- start start 2023-10-12T22:39:56.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.909+0000 7fcce0280700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd8060280 0x7fccd81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:56.912 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.909+0000 7fcce0280700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd8106bd0 con 0x7fccd8060280 2023-10-12T22:39:56.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.909+0000 7fccde01c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd8060280 0x7fccd81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:56.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.909+0000 7fccde01c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd8060280 0x7fccd81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:60748/0 (socket says 172.21.15.195:60748) 2023-10-12T22:39:56.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.909+0000 7fccde01c700 1 -- 172.21.15.195:0/2659511753 learned_addr learned my addr 172.21.15.195:0/2659511753 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:56.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.909+0000 7fccde01c700 1 -- 172.21.15.195:0/2659511753 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd4005040 con 0x7fccd8060280 2023-10-12T22:39:56.913 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.909+0000 7fccde01c700 1 --2- 172.21.15.195:0/2659511753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd8060280 0x7fccd81066e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fccd4013510 tx=0x7fccd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:56.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.910+0000 7fccceffd700 1 -- 172.21.15.195:0/2659511753 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccd4009320 con 0x7fccd8060280 2023-10-12T22:39:56.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.910+0000 7fcce0280700 1 -- 172.21.15.195:0/2659511753 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd8106dd0 con 0x7fccd8060280 2023-10-12T22:39:56.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.910+0000 7fccceffd700 1 -- 172.21.15.195:0/2659511753 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccd4013f00 con 0x7fccd8060280 2023-10-12T22:39:56.914 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.910+0000 7fccceffd700 1 -- 172.21.15.195:0/2659511753 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccd4014dc0 con 0x7fccd8060280 2023-10-12T22:39:56.915 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.910+0000 7fcce0280700 1 -- 172.21.15.195:0/2659511753 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd8107260 con 0x7fccd8060280 2023-10-12T22:39:56.916 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.910+0000 7fccceffd700 1 -- 172.21.15.195:0/2659511753 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fccd4014420 con 0x7fccd8060280 2023-10-12T22:39:56.916 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.911+0000 7fccceffd700 1 --2- 172.21.15.195:0/2659511753 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fccc402ff60 0x7fccc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:56.916 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.911+0000 7fcce0280700 1 -- 172.21.15.195:0/2659511753 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd8045ad0 con 0x7fccd8060280 2023-10-12T22:39:56.916 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.911+0000 7fccceffd700 1 -- 172.21.15.195:0/2659511753 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fccd4041030 con 0x7fccd8060280 2023-10-12T22:39:56.916 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.911+0000 7fccdd81b700 1 --2- 172.21.15.195:0/2659511753 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fccc402ff60 0x7fccc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:56.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.911+0000 7fccdd81b700 1 --2- 172.21.15.195:0/2659511753 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fccc402ff60 0x7fccc4032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fccc80096a0 tx=0x7fccc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:56.917 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:56.915+0000 7fccceffd700 1 -- 172.21.15.195:0/2659511753 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fccd40095f0 con 0x7fccd8060280 2023-10-12T22:39:57.120 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:57.120+0000 7fcce0280700 1 -- 172.21.15.195:0/2659511753 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccd8108110 con 0x7fccd8060280 2023-10-12T22:39:57.121 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:57.121+0000 7fccceffd700 1 -- 172.21.15.195:0/2659511753 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fccd401cb10 con 0x7fccd8060280 2023-10-12T22:39:57.121 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:57.121 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:57.122 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:57.122+0000 7fcce0280700 1 -- 172.21.15.195:0/2659511753 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fccc402ff60 msgr2=0x7fccc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:57.123 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:57.122+0000 7fcce0280700 1 --2- 172.21.15.195:0/2659511753 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fccc402ff60 0x7fccc4032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fccc80096a0 tx=0x7fccc8006b40).stop 2023-10-12T22:39:57.123 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:57.122+0000 7fcce0280700 1 -- 172.21.15.195:0/2659511753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd8060280 msgr2=0x7fccd81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:57.123 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:57.122+0000 7fcce0280700 1 --2- 172.21.15.195:0/2659511753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd8060280 0x7fccd81066e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fccd4013510 tx=0x7fccd400b040).stop 2023-10-12T22:39:57.123 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:57.122+0000 7fcce0280700 1 -- 172.21.15.195:0/2659511753 shutdown_connections 2023-10-12T22:39:57.123 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:57.122+0000 7fcce0280700 1 --2- 172.21.15.195:0/2659511753 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fccc402ff60 0x7fccc4032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:57.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:57.122+0000 7fcce0280700 1 --2- 172.21.15.195:0/2659511753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fccd8060280 0x7fccd81066e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:57.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:57.122+0000 7fcce0280700 1 -- 172.21.15.195:0/2659511753 >> 172.21.15.195:0/2659511753 conn(0x7fccd80f0b70 msgr2=0x7fccd80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:57.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:57.122+0000 7fcce0280700 1 -- 172.21.15.195:0/2659511753 shutdown_connections 2023-10-12T22:39:57.124 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:57.122+0000 7fcce0280700 1 -- 172.21.15.195:0/2659511753 wait complete. 2023-10-12T22:39:57.125 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:39:57.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:57 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:57.121197+0000 mon.smithi131 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.195:0/2659511753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:39:58.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:39:58 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:56.878037+0000 mgr.smithi131.efktma (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:39:58.524 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:39:58.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:39:59.632 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.631+0000 7f9684f18700 1 -- 172.21.15.195:0/216941641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96800f4740 msgr2=0x7f96800f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:59.633 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.631+0000 7f9684f18700 1 --2- 172.21.15.195:0/216941641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96800f4740 0x7f96800f4b60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f9674009fb0 tx=0x7f9674009b30).stop 2023-10-12T22:39:59.633 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.631+0000 7f9684f18700 1 -- 172.21.15.195:0/216941641 shutdown_connections 2023-10-12T22:39:59.633 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.631+0000 7f9684f18700 1 --2- 172.21.15.195:0/216941641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96800f4740 0x7f96800f4b60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:59.633 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.631+0000 7f9684f18700 1 -- 172.21.15.195:0/216941641 >> 172.21.15.195:0/216941641 conn(0x7f96800f0260 msgr2=0x7f96800f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:59.634 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.631+0000 7f9684f18700 1 -- 172.21.15.195:0/216941641 shutdown_connections 2023-10-12T22:39:59.634 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.631+0000 7f9684f18700 1 -- 172.21.15.195:0/216941641 wait complete. 2023-10-12T22:39:59.634 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.632+0000 7f9684f18700 1 Processor -- start 2023-10-12T22:39:59.634 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.632+0000 7f9684f18700 1 -- start start 2023-10-12T22:39:59.634 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.632+0000 7f9684f18700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96800f4740 0x7f96801044b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:59.635 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.632+0000 7f9684f18700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96801049a0 con 0x7f96800f4740 2023-10-12T22:39:59.635 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.632+0000 7f967e59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96800f4740 0x7f96801044b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:59.635 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.632+0000 7f967e59c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96800f4740 0x7f96801044b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:47150/0 (socket says 172.21.15.195:47150) 2023-10-12T22:39:59.635 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.632+0000 7f967e59c700 1 -- 172.21.15.195:0/3698944050 learned_addr learned my addr 172.21.15.195:0/3698944050 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:39:59.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.633+0000 7f967e59c700 1 -- 172.21.15.195:0/3698944050 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9674005040 con 0x7f96800f4740 2023-10-12T22:39:59.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.633+0000 7f967e59c700 1 --2- 172.21.15.195:0/3698944050 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96800f4740 0x7f96801044b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f9674014000 tx=0x7f967400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:59.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.633+0000 7f966effd700 1 -- 172.21.15.195:0/3698944050 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96740095f0 con 0x7f96800f4740 2023-10-12T22:39:59.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.633+0000 7f966effd700 1 -- 172.21.15.195:0/3698944050 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9674007e70 con 0x7f96800f4740 2023-10-12T22:39:59.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.633+0000 7f9684f18700 1 -- 172.21.15.195:0/3698944050 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9680104ba0 con 0x7f96800f4740 2023-10-12T22:39:59.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.633+0000 7f966effd700 1 -- 172.21.15.195:0/3698944050 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9674014ea0 con 0x7f96800f4740 2023-10-12T22:39:59.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.633+0000 7f9684f18700 1 -- 172.21.15.195:0/3698944050 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9680105030 con 0x7f96800f4740 2023-10-12T22:39:59.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.634+0000 7f966effd700 1 -- 172.21.15.195:0/3698944050 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9674025070 con 0x7f96800f4740 2023-10-12T22:39:59.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.634+0000 7f9684f18700 1 -- 172.21.15.195:0/3698944050 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9680045ad0 con 0x7f96800f4740 2023-10-12T22:39:59.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.634+0000 7f966effd700 1 --2- 172.21.15.195:0/3698944050 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f96680343b0 0x7f9668036870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:39:59.639 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.634+0000 7f966effd700 1 -- 172.21.15.195:0/3698944050 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96740408e0 con 0x7f96800f4740 2023-10-12T22:39:59.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.634+0000 7f967dd9b700 1 --2- 172.21.15.195:0/3698944050 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f96680343b0 0x7f9668036870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:39:59.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.634+0000 7f967dd9b700 1 --2- 172.21.15.195:0/3698944050 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f96680343b0 0x7f9668036870 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f96700096a0 tx=0x7f9670006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:39:59.640 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.639+0000 7f966effd700 1 -- 172.21.15.195:0/3698944050 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96740079c0 con 0x7f96800f4740 2023-10-12T22:39:59.845 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.844+0000 7f9684f18700 1 -- 172.21.15.195:0/3698944050 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96800f7b10 con 0x7f96800f4740 2023-10-12T22:39:59.845 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.844+0000 7f966effd700 1 -- 172.21.15.195:0/3698944050 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f967400f300 con 0x7f96800f4740 2023-10-12T22:39:59.846 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:39:59.846 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:39:59.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.846+0000 7f9684f18700 1 -- 172.21.15.195:0/3698944050 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f96680343b0 msgr2=0x7f9668036870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:59.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.846+0000 7f9684f18700 1 --2- 172.21.15.195:0/3698944050 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f96680343b0 0x7f9668036870 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f96700096a0 tx=0x7f9670006b40).stop 2023-10-12T22:39:59.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.846+0000 7f9684f18700 1 -- 172.21.15.195:0/3698944050 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96800f4740 msgr2=0x7f96801044b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:39:59.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.846+0000 7f9684f18700 1 --2- 172.21.15.195:0/3698944050 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96800f4740 0x7f96801044b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f9674014000 tx=0x7f967400b040).stop 2023-10-12T22:39:59.847 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.846+0000 7f9684f18700 1 -- 172.21.15.195:0/3698944050 shutdown_connections 2023-10-12T22:39:59.848 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.846+0000 7f9684f18700 1 --2- 172.21.15.195:0/3698944050 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f96680343b0 0x7f9668036870 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:59.848 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.846+0000 7f9684f18700 1 --2- 172.21.15.195:0/3698944050 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96800f4740 0x7f96801044b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:39:59.848 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.846+0000 7f9684f18700 1 -- 172.21.15.195:0/3698944050 >> 172.21.15.195:0/3698944050 conn(0x7f96800f0260 msgr2=0x7f96800fb2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:39:59.848 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.846+0000 7f9684f18700 1 -- 172.21.15.195:0/3698944050 shutdown_connections 2023-10-12T22:39:59.848 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:39:59.846+0000 7f9684f18700 1 -- 172.21.15.195:0/3698944050 wait complete. 2023-10-12T22:39:59.849 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:40:00.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:39:58.878315+0000 mgr.smithi131.efktma (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:40:00.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:00 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:39:59.844995+0000 mon.smithi131 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.195:0/3698944050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:40:01.213 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:40:01.213 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:40:02.354 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.352+0000 7f3d3d738700 1 -- 172.21.15.195:0/172708256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d380f6510 msgr2=0x7f3d380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:02.354 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.352+0000 7f3d3d738700 1 --2- 172.21.15.195:0/172708256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d380f6510 0x7f3d380f6930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f3d20009fb0 tx=0x7f3d20009b30).stop 2023-10-12T22:40:02.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.352+0000 7f3d3d738700 1 -- 172.21.15.195:0/172708256 shutdown_connections 2023-10-12T22:40:02.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.352+0000 7f3d3d738700 1 --2- 172.21.15.195:0/172708256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d380f6510 0x7f3d380f6930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:02.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.352+0000 7f3d3d738700 1 -- 172.21.15.195:0/172708256 >> 172.21.15.195:0/172708256 conn(0x7f3d380f1c70 msgr2=0x7f3d380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:02.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.352+0000 7f3d3d738700 1 -- 172.21.15.195:0/172708256 shutdown_connections 2023-10-12T22:40:02.355 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.352+0000 7f3d3d738700 1 -- 172.21.15.195:0/172708256 wait complete. 2023-10-12T22:40:02.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.353+0000 7f3d3d738700 1 Processor -- start 2023-10-12T22:40:02.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.353+0000 7f3d3d738700 1 -- start start 2023-10-12T22:40:02.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.353+0000 7f3d3d738700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d380f6510 0x7f3d3806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:02.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.353+0000 7f3d3d738700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d38069e20 con 0x7f3d380f6510 2023-10-12T22:40:02.356 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.353+0000 7f3d36ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d380f6510 0x7f3d3806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:02.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.353+0000 7f3d36ffd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d380f6510 0x7f3d3806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:47172/0 (socket says 172.21.15.195:47172) 2023-10-12T22:40:02.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.354+0000 7f3d36ffd700 1 -- 172.21.15.195:0/3527447347 learned_addr learned my addr 172.21.15.195:0/3527447347 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:40:02.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.354+0000 7f3d36ffd700 1 -- 172.21.15.195:0/3527447347 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d20005040 con 0x7f3d380f6510 2023-10-12T22:40:02.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.354+0000 7f3d36ffd700 1 --2- 172.21.15.195:0/3527447347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d380f6510 0x7f3d3806b9a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f3d20013510 tx=0x7f3d2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:02.357 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.354+0000 7f3d2f7fe700 1 -- 172.21.15.195:0/3527447347 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d20009320 con 0x7f3d380f6510 2023-10-12T22:40:02.358 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.354+0000 7f3d3d738700 1 -- 172.21.15.195:0/3527447347 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d38069fc0 con 0x7f3d380f6510 2023-10-12T22:40:02.358 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.354+0000 7f3d2f7fe700 1 -- 172.21.15.195:0/3527447347 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d20013f00 con 0x7f3d380f6510 2023-10-12T22:40:02.358 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.354+0000 7f3d2f7fe700 1 -- 172.21.15.195:0/3527447347 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d20014d90 con 0x7f3d380f6510 2023-10-12T22:40:02.358 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.354+0000 7f3d3d738700 1 -- 172.21.15.195:0/3527447347 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3806a450 con 0x7f3d380f6510 2023-10-12T22:40:02.359 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.355+0000 7f3d2f7fe700 1 -- 172.21.15.195:0/3527447347 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d200095f0 con 0x7f3d380f6510 2023-10-12T22:40:02.359 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.355+0000 7f3d3d738700 1 -- 172.21.15.195:0/3527447347 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d38059100 con 0x7f3d380f6510 2023-10-12T22:40:02.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.355+0000 7f3d2f7fe700 1 --2- 172.21.15.195:0/3527447347 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3d2402ff60 0x7f3d24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:02.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.355+0000 7f3d2f7fe700 1 -- 172.21.15.195:0/3527447347 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d200410a0 con 0x7f3d380f6510 2023-10-12T22:40:02.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.355+0000 7f3d367fc700 1 --2- 172.21.15.195:0/3527447347 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3d2402ff60 0x7f3d24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:02.360 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.356+0000 7f3d367fc700 1 --2- 172.21.15.195:0/3527447347 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3d2402ff60 0x7f3d24032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3d280096a0 tx=0x7f3d28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:02.361 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.360+0000 7f3d2f7fe700 1 -- 172.21.15.195:0/3527447347 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d2001d7f0 con 0x7f3d380f6510 2023-10-12T22:40:02.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:02 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:40:00.878690+0000 mgr.smithi131.efktma (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:40:02.566 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.565+0000 7f3d3d738700 1 -- 172.21.15.195:0/3527447347 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d380fbf60 con 0x7f3d380f6510 2023-10-12T22:40:02.566 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.566+0000 7f3d2f7fe700 1 -- 172.21.15.195:0/3527447347 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3d20010230 con 0x7f3d380f6510 2023-10-12T22:40:02.567 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:40:02.567 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:40:02.568 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.567+0000 7f3d3d738700 1 -- 172.21.15.195:0/3527447347 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3d2402ff60 msgr2=0x7f3d24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:02.568 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.567+0000 7f3d3d738700 1 --2- 172.21.15.195:0/3527447347 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3d2402ff60 0x7f3d24032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3d280096a0 tx=0x7f3d28006b40).stop 2023-10-12T22:40:02.568 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.567+0000 7f3d3d738700 1 -- 172.21.15.195:0/3527447347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d380f6510 msgr2=0x7f3d3806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:02.568 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.567+0000 7f3d3d738700 1 --2- 172.21.15.195:0/3527447347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d380f6510 0x7f3d3806b9a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f3d20013510 tx=0x7f3d2000b040).stop 2023-10-12T22:40:02.569 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.568+0000 7f3d3d738700 1 -- 172.21.15.195:0/3527447347 shutdown_connections 2023-10-12T22:40:02.569 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.568+0000 7f3d3d738700 1 --2- 172.21.15.195:0/3527447347 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f3d2402ff60 0x7f3d24032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:02.569 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.568+0000 7f3d3d738700 1 --2- 172.21.15.195:0/3527447347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d380f6510 0x7f3d3806b9a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:02.569 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.568+0000 7f3d3d738700 1 -- 172.21.15.195:0/3527447347 >> 172.21.15.195:0/3527447347 conn(0x7f3d380f1c70 msgr2=0x7f3d380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:02.569 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.568+0000 7f3d3d738700 1 -- 172.21.15.195:0/3527447347 shutdown_connections 2023-10-12T22:40:02.570 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:02.568+0000 7f3d3d738700 1 -- 172.21.15.195:0/3527447347 wait complete. 2023-10-12T22:40:02.570 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:40:03.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:03 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:40:02.566571+0000 mon.smithi131 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.195:0/3527447347' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:40:03.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:40:03.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:40:04.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:04 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:40:02.878988+0000 mgr.smithi131.efktma (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:40:05.108 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.105+0000 7fc05b569700 1 -- 172.21.15.195:0/3218089427 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0540f6510 msgr2=0x7fc0540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:05.108 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.105+0000 7fc05b569700 1 --2- 172.21.15.195:0/3218089427 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0540f6510 0x7fc0540f6930 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fc044009fb0 tx=0x7fc044009b30).stop 2023-10-12T22:40:05.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.105+0000 7fc05b569700 1 -- 172.21.15.195:0/3218089427 shutdown_connections 2023-10-12T22:40:05.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.105+0000 7fc05b569700 1 --2- 172.21.15.195:0/3218089427 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0540f6510 0x7fc0540f6930 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:05.109 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.105+0000 7fc05b569700 1 -- 172.21.15.195:0/3218089427 >> 172.21.15.195:0/3218089427 conn(0x7fc0540f1c70 msgr2=0x7fc0540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:05.110 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.105+0000 7fc05b569700 1 -- 172.21.15.195:0/3218089427 shutdown_connections 2023-10-12T22:40:05.110 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.105+0000 7fc05b569700 1 -- 172.21.15.195:0/3218089427 wait complete. 2023-10-12T22:40:05.110 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.106+0000 7fc05b569700 1 Processor -- start 2023-10-12T22:40:05.110 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.106+0000 7fc05b569700 1 -- start start 2023-10-12T22:40:05.111 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.106+0000 7fc05b569700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0540f6510 0x7fc054106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:05.111 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.106+0000 7fc05b569700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc054106c20 con 0x7fc0540f6510 2023-10-12T22:40:05.111 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.106+0000 7fc059305700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0540f6510 0x7fc054106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:05.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.106+0000 7fc059305700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0540f6510 0x7fc054106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:47202/0 (socket says 172.21.15.195:47202) 2023-10-12T22:40:05.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.106+0000 7fc059305700 1 -- 172.21.15.195:0/2209988991 learned_addr learned my addr 172.21.15.195:0/2209988991 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:40:05.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.106+0000 7fc059305700 1 -- 172.21.15.195:0/2209988991 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc044005040 con 0x7fc0540f6510 2023-10-12T22:40:05.112 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.107+0000 7fc059305700 1 --2- 172.21.15.195:0/2209988991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0540f6510 0x7fc054106730 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fc044013510 tx=0x7fc04400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:05.113 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.107+0000 7fc049ffb700 1 -- 172.21.15.195:0/2209988991 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc044009320 con 0x7fc0540f6510 2023-10-12T22:40:05.113 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.107+0000 7fc05b569700 1 -- 172.21.15.195:0/2209988991 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc054106e20 con 0x7fc0540f6510 2023-10-12T22:40:05.114 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.107+0000 7fc049ffb700 1 -- 172.21.15.195:0/2209988991 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc044013f00 con 0x7fc0540f6510 2023-10-12T22:40:05.114 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.107+0000 7fc049ffb700 1 -- 172.21.15.195:0/2209988991 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc044014d90 con 0x7fc0540f6510 2023-10-12T22:40:05.114 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.107+0000 7fc05b569700 1 -- 172.21.15.195:0/2209988991 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0541072b0 con 0x7fc0540f6510 2023-10-12T22:40:05.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.108+0000 7fc05b569700 1 -- 172.21.15.195:0/2209988991 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc054045ad0 con 0x7fc0540f6510 2023-10-12T22:40:05.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.108+0000 7fc049ffb700 1 -- 172.21.15.195:0/2209988991 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0440095f0 con 0x7fc0540f6510 2023-10-12T22:40:05.115 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.108+0000 7fc049ffb700 1 --2- 172.21.15.195:0/2209988991 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc04002ff60 0x7fc040032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:05.116 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.108+0000 7fc049ffb700 1 -- 172.21.15.195:0/2209988991 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0440410c0 con 0x7fc0540f6510 2023-10-12T22:40:05.116 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.108+0000 7fc058b04700 1 --2- 172.21.15.195:0/2209988991 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc04002ff60 0x7fc040032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:05.116 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.109+0000 7fc058b04700 1 --2- 172.21.15.195:0/2209988991 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc04002ff60 0x7fc040032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fc0500096a0 tx=0x7fc050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:05.116 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.113+0000 7fc049ffb700 1 -- 172.21.15.195:0/2209988991 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc044014420 con 0x7fc0540f6510 2023-10-12T22:40:05.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.324+0000 7fc05b569700 1 -- 172.21.15.195:0/2209988991 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0540fbf60 con 0x7fc0540f6510 2023-10-12T22:40:05.325 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.324+0000 7fc049ffb700 1 -- 172.21.15.195:0/2209988991 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc04401d7f0 con 0x7fc0540f6510 2023-10-12T22:40:05.326 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:40:05.326 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:40:05.327 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.326+0000 7fc05b569700 1 -- 172.21.15.195:0/2209988991 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc04002ff60 msgr2=0x7fc040032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:05.327 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.326+0000 7fc05b569700 1 --2- 172.21.15.195:0/2209988991 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc04002ff60 0x7fc040032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fc0500096a0 tx=0x7fc050006b40).stop 2023-10-12T22:40:05.327 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.326+0000 7fc05b569700 1 -- 172.21.15.195:0/2209988991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0540f6510 msgr2=0x7fc054106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:05.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.326+0000 7fc05b569700 1 --2- 172.21.15.195:0/2209988991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0540f6510 0x7fc054106730 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fc044013510 tx=0x7fc04400b040).stop 2023-10-12T22:40:05.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.326+0000 7fc05b569700 1 -- 172.21.15.195:0/2209988991 shutdown_connections 2023-10-12T22:40:05.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.326+0000 7fc05b569700 1 --2- 172.21.15.195:0/2209988991 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc04002ff60 0x7fc040032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:05.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.326+0000 7fc05b569700 1 --2- 172.21.15.195:0/2209988991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0540f6510 0x7fc054106730 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:05.328 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.326+0000 7fc05b569700 1 -- 172.21.15.195:0/2209988991 >> 172.21.15.195:0/2209988991 conn(0x7fc0540f1c70 msgr2=0x7fc0540f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:05.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.326+0000 7fc05b569700 1 -- 172.21.15.195:0/2209988991 shutdown_connections 2023-10-12T22:40:05.329 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:05.326+0000 7fc05b569700 1 -- 172.21.15.195:0/2209988991 wait complete. 2023-10-12T22:40:05.329 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:40:06.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:40:04.879391+0000 mgr.smithi131.efktma (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:40:06.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:06 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:40:05.325757+0000 mon.smithi131 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.195:0/2209988991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:40:06.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:40:06.745 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:40:07.926 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.925+0000 7fa53c41e700 1 -- 172.21.15.195:0/4259472631 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5340f5410 msgr2=0x7fa5340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:07.927 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.925+0000 7fa53c41e700 1 --2- 172.21.15.195:0/4259472631 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5340f5410 0x7fa5340f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fa528009fb0 tx=0x7fa528009b30).stop 2023-10-12T22:40:07.927 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.925+0000 7fa53c41e700 1 -- 172.21.15.195:0/4259472631 shutdown_connections 2023-10-12T22:40:07.927 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.925+0000 7fa53c41e700 1 --2- 172.21.15.195:0/4259472631 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5340f5410 0x7fa5340f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:07.927 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.925+0000 7fa53c41e700 1 -- 172.21.15.195:0/4259472631 >> 172.21.15.195:0/4259472631 conn(0x7fa5340f0b70 msgr2=0x7fa5340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:07.928 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.925+0000 7fa53c41e700 1 -- 172.21.15.195:0/4259472631 shutdown_connections 2023-10-12T22:40:07.928 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.925+0000 7fa53c41e700 1 -- 172.21.15.195:0/4259472631 wait complete. 2023-10-12T22:40:07.930 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.926+0000 7fa53c41e700 1 Processor -- start 2023-10-12T22:40:07.930 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.926+0000 7fa53c41e700 1 -- start start 2023-10-12T22:40:07.930 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.926+0000 7fa53c41e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5340f5410 0x7fa534108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:07.930 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.926+0000 7fa53c41e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa534108e10 con 0x7fa5340f5410 2023-10-12T22:40:07.930 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.926+0000 7fa53a1ba700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5340f5410 0x7fa534108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:07.931 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.926+0000 7fa53a1ba700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5340f5410 0x7fa534108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:47214/0 (socket says 172.21.15.195:47214) 2023-10-12T22:40:07.931 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.926+0000 7fa53a1ba700 1 -- 172.21.15.195:0/2992735453 learned_addr learned my addr 172.21.15.195:0/2992735453 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:40:07.931 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.926+0000 7fa53a1ba700 1 -- 172.21.15.195:0/2992735453 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa528005040 con 0x7fa5340f5410 2023-10-12T22:40:07.931 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.927+0000 7fa53a1ba700 1 --2- 172.21.15.195:0/2992735453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5340f5410 0x7fa534108920 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fa528013510 tx=0x7fa52800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:07.931 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.927+0000 7fa526ffd700 1 -- 172.21.15.195:0/2992735453 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa528009320 con 0x7fa5340f5410 2023-10-12T22:40:07.932 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.927+0000 7fa526ffd700 1 -- 172.21.15.195:0/2992735453 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa528013f00 con 0x7fa5340f5410 2023-10-12T22:40:07.932 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.927+0000 7fa53c41e700 1 -- 172.21.15.195:0/2992735453 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa534109010 con 0x7fa5340f5410 2023-10-12T22:40:07.932 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.927+0000 7fa526ffd700 1 -- 172.21.15.195:0/2992735453 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa528014d90 con 0x7fa5340f5410 2023-10-12T22:40:07.932 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.927+0000 7fa53c41e700 1 -- 172.21.15.195:0/2992735453 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5341094a0 con 0x7fa5340f5410 2023-10-12T22:40:07.932 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.928+0000 7fa526ffd700 1 -- 172.21.15.195:0/2992735453 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa528014420 con 0x7fa5340f5410 2023-10-12T22:40:07.933 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.928+0000 7fa53c41e700 1 -- 172.21.15.195:0/2992735453 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa534045ad0 con 0x7fa5340f5410 2023-10-12T22:40:07.933 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.928+0000 7fa526ffd700 1 --2- 172.21.15.195:0/2992735453 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa52002ff60 0x7fa520032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:07.933 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.928+0000 7fa526ffd700 1 -- 172.21.15.195:0/2992735453 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa528041000 con 0x7fa5340f5410 2023-10-12T22:40:07.933 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.928+0000 7fa5399b9700 1 --2- 172.21.15.195:0/2992735453 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa52002ff60 0x7fa520032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:07.933 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.928+0000 7fa5399b9700 1 --2- 172.21.15.195:0/2992735453 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa52002ff60 0x7fa520032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fa5300096a0 tx=0x7fa530006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:07.934 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:07.932+0000 7fa526ffd700 1 -- 172.21.15.195:0/2992735453 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5280095f0 con 0x7fa5340f5410 2023-10-12T22:40:08.138 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:08.137+0000 7fa53c41e700 1 -- 172.21.15.195:0/2992735453 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa53410a350 con 0x7fa5340f5410 2023-10-12T22:40:08.139 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:08.138+0000 7fa526ffd700 1 -- 172.21.15.195:0/2992735453 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa52801db70 con 0x7fa5340f5410 2023-10-12T22:40:08.140 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:40:08.140 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:40:08.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:08.139+0000 7fa53c41e700 1 -- 172.21.15.195:0/2992735453 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa52002ff60 msgr2=0x7fa520032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:08.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:08.139+0000 7fa53c41e700 1 --2- 172.21.15.195:0/2992735453 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa52002ff60 0x7fa520032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fa5300096a0 tx=0x7fa530006b40).stop 2023-10-12T22:40:08.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:08.139+0000 7fa53c41e700 1 -- 172.21.15.195:0/2992735453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5340f5410 msgr2=0x7fa534108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:08.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:08.139+0000 7fa53c41e700 1 --2- 172.21.15.195:0/2992735453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5340f5410 0x7fa534108920 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fa528013510 tx=0x7fa52800b040).stop 2023-10-12T22:40:08.141 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:08.140+0000 7fa53c41e700 1 -- 172.21.15.195:0/2992735453 shutdown_connections 2023-10-12T22:40:08.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:08.140+0000 7fa53c41e700 1 --2- 172.21.15.195:0/2992735453 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fa52002ff60 0x7fa520032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:08.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:08.140+0000 7fa53c41e700 1 --2- 172.21.15.195:0/2992735453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5340f5410 0x7fa534108920 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:08.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:08.140+0000 7fa53c41e700 1 -- 172.21.15.195:0/2992735453 >> 172.21.15.195:0/2992735453 conn(0x7fa5340f0b70 msgr2=0x7fa53405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:08.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:08.140+0000 7fa53c41e700 1 -- 172.21.15.195:0/2992735453 shutdown_connections 2023-10-12T22:40:08.142 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:08.140+0000 7fa53c41e700 1 -- 172.21.15.195:0/2992735453 wait complete. 2023-10-12T22:40:08.143 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:40:08.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:40:06.879781+0000 mgr.smithi131.efktma (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:40:08.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:08 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:40:08.139585+0000 mon.smithi131 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.195:0/2992735453' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:40:09.520 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:40:09.521 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:40:10.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:10 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:40:08.880215+0000 mgr.smithi131.efktma (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:40:10.658 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.656+0000 7f684a92b700 1 -- 172.21.15.195:0/1152444762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68440f62a0 msgr2=0x7f68440f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:10.658 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.656+0000 7f684a92b700 1 --2- 172.21.15.195:0/1152444762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68440f62a0 0x7f68440f66c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f6834009fb0 tx=0x7f6834009b30).stop 2023-10-12T22:40:10.659 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.656+0000 7f684a92b700 1 -- 172.21.15.195:0/1152444762 shutdown_connections 2023-10-12T22:40:10.659 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.656+0000 7f684a92b700 1 --2- 172.21.15.195:0/1152444762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68440f62a0 0x7f68440f66c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:10.659 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.656+0000 7f684a92b700 1 -- 172.21.15.195:0/1152444762 >> 172.21.15.195:0/1152444762 conn(0x7f68440f1ac0 msgr2=0x7f68440f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:10.659 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.656+0000 7f684a92b700 1 -- 172.21.15.195:0/1152444762 shutdown_connections 2023-10-12T22:40:10.659 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.656+0000 7f684a92b700 1 -- 172.21.15.195:0/1152444762 wait complete. 2023-10-12T22:40:10.660 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.656+0000 7f684a92b700 1 Processor -- start 2023-10-12T22:40:10.660 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.657+0000 7f684a92b700 1 -- start start 2023-10-12T22:40:10.660 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.657+0000 7f684a92b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68440f62a0 0x7f68441064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:10.660 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.657+0000 7f684a92b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6844106990 con 0x7f68440f62a0 2023-10-12T22:40:10.660 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.657+0000 7f6849929700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68440f62a0 0x7f68441064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:10.661 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.657+0000 7f6849929700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68440f62a0 0x7f68441064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:57172/0 (socket says 172.21.15.195:57172) 2023-10-12T22:40:10.661 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.657+0000 7f6849929700 1 -- 172.21.15.195:0/315540870 learned_addr learned my addr 172.21.15.195:0/315540870 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:40:10.662 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.657+0000 7f6849929700 1 -- 172.21.15.195:0/315540870 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6834005040 con 0x7f68440f62a0 2023-10-12T22:40:10.663 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.657+0000 7f6849929700 1 --2- 172.21.15.195:0/315540870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68440f62a0 0x7f68441064a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f6834013510 tx=0x7f683400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:10.663 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.658+0000 7f683a7fc700 1 -- 172.21.15.195:0/315540870 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6834009320 con 0x7f68440f62a0 2023-10-12T22:40:10.663 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.658+0000 7f684a92b700 1 -- 172.21.15.195:0/315540870 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6844106b90 con 0x7f68440f62a0 2023-10-12T22:40:10.663 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.658+0000 7f683a7fc700 1 -- 172.21.15.195:0/315540870 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6834013f00 con 0x7f68440f62a0 2023-10-12T22:40:10.663 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.658+0000 7f683a7fc700 1 -- 172.21.15.195:0/315540870 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6834014dc0 con 0x7f68440f62a0 2023-10-12T22:40:10.664 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.658+0000 7f684a92b700 1 -- 172.21.15.195:0/315540870 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6844107020 con 0x7f68440f62a0 2023-10-12T22:40:10.664 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.658+0000 7f683a7fc700 1 -- 172.21.15.195:0/315540870 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68340095f0 con 0x7f68440f62a0 2023-10-12T22:40:10.664 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.658+0000 7f684a92b700 1 -- 172.21.15.195:0/315540870 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6844045ad0 con 0x7f68440f62a0 2023-10-12T22:40:10.664 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.659+0000 7f683a7fc700 1 --2- 172.21.15.195:0/315540870 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f683002ff60 0x7f6830032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:10.664 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.659+0000 7f683a7fc700 1 -- 172.21.15.195:0/315540870 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68340410c0 con 0x7f68440f62a0 2023-10-12T22:40:10.665 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.659+0000 7f6849128700 1 --2- 172.21.15.195:0/315540870 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f683002ff60 0x7f6830032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:10.665 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.659+0000 7f6849128700 1 --2- 172.21.15.195:0/315540870 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f683002ff60 0x7f6830032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f68400096a0 tx=0x7f6840006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:10.665 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.663+0000 7f683a7fc700 1 -- 172.21.15.195:0/315540870 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6834014420 con 0x7f68440f62a0 2023-10-12T22:40:10.871 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.870+0000 7f684a92b700 1 -- 172.21.15.195:0/315540870 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6844107ed0 con 0x7f68440f62a0 2023-10-12T22:40:10.872 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.871+0000 7f683a7fc700 1 -- 172.21.15.195:0/315540870 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6834010230 con 0x7f68440f62a0 2023-10-12T22:40:10.872 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:40:10.873 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:40:10.873 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.872+0000 7f684a92b700 1 -- 172.21.15.195:0/315540870 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f683002ff60 msgr2=0x7f6830032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:10.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.872+0000 7f684a92b700 1 --2- 172.21.15.195:0/315540870 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f683002ff60 0x7f6830032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f68400096a0 tx=0x7f6840006b40).stop 2023-10-12T22:40:10.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.872+0000 7f684a92b700 1 -- 172.21.15.195:0/315540870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68440f62a0 msgr2=0x7f68441064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:10.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.872+0000 7f684a92b700 1 --2- 172.21.15.195:0/315540870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68440f62a0 0x7f68441064a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f6834013510 tx=0x7f683400b040).stop 2023-10-12T22:40:10.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.872+0000 7f684a92b700 1 -- 172.21.15.195:0/315540870 shutdown_connections 2023-10-12T22:40:10.874 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.872+0000 7f684a92b700 1 --2- 172.21.15.195:0/315540870 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f683002ff60 0x7f6830032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:10.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.872+0000 7f684a92b700 1 --2- 172.21.15.195:0/315540870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68440f62a0 0x7f68441064a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:10.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.872+0000 7f684a92b700 1 -- 172.21.15.195:0/315540870 >> 172.21.15.195:0/315540870 conn(0x7f68440f1ac0 msgr2=0x7f68440f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:10.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.872+0000 7f684a92b700 1 -- 172.21.15.195:0/315540870 shutdown_connections 2023-10-12T22:40:10.875 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:10.872+0000 7f684a92b700 1 -- 172.21.15.195:0/315540870 wait complete. 2023-10-12T22:40:10.876 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:40:11.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:11 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:40:10.872262+0000 mon.smithi131 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.195:0/315540870' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:40:12.259 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:40:12.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:40:12.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:12 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:40:10.880605+0000 mgr.smithi131.efktma (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:40:13.423 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.420+0000 7fc0ff807700 1 -- 172.21.15.195:0/350326985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0f80f6510 msgr2=0x7fc0f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:13.423 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.420+0000 7fc0ff807700 1 --2- 172.21.15.195:0/350326985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0f80f6510 0x7fc0f80f6930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fc0e8009fb0 tx=0x7fc0e8009b30).stop 2023-10-12T22:40:13.423 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.420+0000 7fc0ff807700 1 -- 172.21.15.195:0/350326985 shutdown_connections 2023-10-12T22:40:13.424 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.420+0000 7fc0ff807700 1 --2- 172.21.15.195:0/350326985 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0f80f6510 0x7fc0f80f6930 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:13.424 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.420+0000 7fc0ff807700 1 -- 172.21.15.195:0/350326985 >> 172.21.15.195:0/350326985 conn(0x7fc0f80f1c70 msgr2=0x7fc0f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:13.424 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.420+0000 7fc0ff807700 1 -- 172.21.15.195:0/350326985 shutdown_connections 2023-10-12T22:40:13.424 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.420+0000 7fc0ff807700 1 -- 172.21.15.195:0/350326985 wait complete. 2023-10-12T22:40:13.425 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.420+0000 7fc0ff807700 1 Processor -- start 2023-10-12T22:40:13.425 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.420+0000 7fc0ff807700 1 -- start start 2023-10-12T22:40:13.425 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.421+0000 7fc0ff807700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0f80f6510 0x7fc0f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:13.426 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.421+0000 7fc0ff807700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f8069e20 con 0x7fc0f80f6510 2023-10-12T22:40:13.426 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.421+0000 7fc0fd5a3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0f80f6510 0x7fc0f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:13.426 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.421+0000 7fc0fd5a3700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0f80f6510 0x7fc0f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:57202/0 (socket says 172.21.15.195:57202) 2023-10-12T22:40:13.426 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.421+0000 7fc0fd5a3700 1 -- 172.21.15.195:0/1232852489 learned_addr learned my addr 172.21.15.195:0/1232852489 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:40:13.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.421+0000 7fc0fd5a3700 1 -- 172.21.15.195:0/1232852489 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e8005040 con 0x7fc0f80f6510 2023-10-12T22:40:13.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.421+0000 7fc0fd5a3700 1 --2- 172.21.15.195:0/1232852489 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0f80f6510 0x7fc0f806b9a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fc0e8014000 tx=0x7fc0e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:13.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.421+0000 7fc0edffb700 1 -- 172.21.15.195:0/1232852489 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0e80095f0 con 0x7fc0f80f6510 2023-10-12T22:40:13.427 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.421+0000 7fc0edffb700 1 -- 172.21.15.195:0/1232852489 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0e8007e70 con 0x7fc0f80f6510 2023-10-12T22:40:13.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.421+0000 7fc0ff807700 1 -- 172.21.15.195:0/1232852489 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f8069fc0 con 0x7fc0f80f6510 2023-10-12T22:40:13.428 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.421+0000 7fc0edffb700 1 -- 172.21.15.195:0/1232852489 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0e8014ea0 con 0x7fc0f80f6510 2023-10-12T22:40:13.429 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.422+0000 7fc0ff807700 1 -- 172.21.15.195:0/1232852489 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f806a450 con 0x7fc0f80f6510 2023-10-12T22:40:13.430 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.422+0000 7fc0ff807700 1 -- 172.21.15.195:0/1232852489 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f8059100 con 0x7fc0f80f6510 2023-10-12T22:40:13.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.423+0000 7fc0edffb700 1 -- 172.21.15.195:0/1232852489 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0e8025070 con 0x7fc0f80f6510 2023-10-12T22:40:13.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.423+0000 7fc0edffb700 1 --2- 172.21.15.195:0/1232852489 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc0e402ff60 0x7fc0e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:13.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.423+0000 7fc0edffb700 1 -- 172.21.15.195:0/1232852489 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0e80409f0 con 0x7fc0f80f6510 2023-10-12T22:40:13.431 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.423+0000 7fc0fcda2700 1 --2- 172.21.15.195:0/1232852489 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc0e402ff60 0x7fc0e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:13.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.423+0000 7fc0fcda2700 1 --2- 172.21.15.195:0/1232852489 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc0e402ff60 0x7fc0e4032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fc0f40096a0 tx=0x7fc0f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:13.432 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.427+0000 7fc0edffb700 1 -- 172.21.15.195:0/1232852489 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0e80079c0 con 0x7fc0f80f6510 2023-10-12T22:40:13.634 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.633+0000 7fc0ff807700 1 -- 172.21.15.195:0/1232852489 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0f80fbf60 con 0x7fc0f80f6510 2023-10-12T22:40:13.635 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.634+0000 7fc0edffb700 1 -- 172.21.15.195:0/1232852489 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc0e800f300 con 0x7fc0f80f6510 2023-10-12T22:40:13.635 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:40:13.636 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:40:13.636 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.635+0000 7fc0ff807700 1 -- 172.21.15.195:0/1232852489 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc0e402ff60 msgr2=0x7fc0e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:13.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.635+0000 7fc0ff807700 1 --2- 172.21.15.195:0/1232852489 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc0e402ff60 0x7fc0e4032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fc0f40096a0 tx=0x7fc0f4006b40).stop 2023-10-12T22:40:13.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.635+0000 7fc0ff807700 1 -- 172.21.15.195:0/1232852489 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0f80f6510 msgr2=0x7fc0f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:13.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.635+0000 7fc0ff807700 1 --2- 172.21.15.195:0/1232852489 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0f80f6510 0x7fc0f806b9a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fc0e8014000 tx=0x7fc0e800b040).stop 2023-10-12T22:40:13.637 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.635+0000 7fc0ff807700 1 -- 172.21.15.195:0/1232852489 shutdown_connections 2023-10-12T22:40:13.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.635+0000 7fc0ff807700 1 --2- 172.21.15.195:0/1232852489 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fc0e402ff60 0x7fc0e4032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:13.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.635+0000 7fc0ff807700 1 --2- 172.21.15.195:0/1232852489 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0f80f6510 0x7fc0f806b9a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:13.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.635+0000 7fc0ff807700 1 -- 172.21.15.195:0/1232852489 >> 172.21.15.195:0/1232852489 conn(0x7fc0f80f1c70 msgr2=0x7fc0f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:13.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.635+0000 7fc0ff807700 1 -- 172.21.15.195:0/1232852489 shutdown_connections 2023-10-12T22:40:13.638 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:13.636+0000 7fc0ff807700 1 -- 172.21.15.195:0/1232852489 wait complete. 2023-10-12T22:40:13.639 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:40:14.490 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:14 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:40:12.881002+0000 mgr.smithi131.efktma (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:40:14.491 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:14 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:40:13.635386+0000 mon.smithi131 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.195:0/1232852489' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:40:14.968 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:40:14.969 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:40:16.125 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.123+0000 7fe115730700 1 -- 172.21.15.195:0/1886450757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1100f6510 msgr2=0x7fe1100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:16.125 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.123+0000 7fe115730700 1 --2- 172.21.15.195:0/1886450757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1100f6510 0x7fe1100f6930 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fe104009fb0 tx=0x7fe104009b30).stop 2023-10-12T22:40:16.125 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.123+0000 7fe115730700 1 -- 172.21.15.195:0/1886450757 shutdown_connections 2023-10-12T22:40:16.126 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.123+0000 7fe115730700 1 --2- 172.21.15.195:0/1886450757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1100f6510 0x7fe1100f6930 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:16.126 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.123+0000 7fe115730700 1 -- 172.21.15.195:0/1886450757 >> 172.21.15.195:0/1886450757 conn(0x7fe1100f1c70 msgr2=0x7fe1100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:16.126 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.123+0000 7fe115730700 1 -- 172.21.15.195:0/1886450757 shutdown_connections 2023-10-12T22:40:16.126 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.123+0000 7fe115730700 1 -- 172.21.15.195:0/1886450757 wait complete. 2023-10-12T22:40:16.129 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.124+0000 7fe115730700 1 Processor -- start 2023-10-12T22:40:16.129 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.124+0000 7fe115730700 1 -- start start 2023-10-12T22:40:16.129 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.124+0000 7fe115730700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1100f6510 0x7fe110106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:16.130 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.124+0000 7fe115730700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe110106c20 con 0x7fe1100f6510 2023-10-12T22:40:16.130 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.125+0000 7fe10effd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1100f6510 0x7fe110106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:16.130 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.125+0000 7fe10effd700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1100f6510 0x7fe110106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:57214/0 (socket says 172.21.15.195:57214) 2023-10-12T22:40:16.131 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.125+0000 7fe10effd700 1 -- 172.21.15.195:0/3155887906 learned_addr learned my addr 172.21.15.195:0/3155887906 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:40:16.131 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.125+0000 7fe10effd700 1 -- 172.21.15.195:0/3155887906 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe104005040 con 0x7fe1100f6510 2023-10-12T22:40:16.131 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.125+0000 7fe10effd700 1 --2- 172.21.15.195:0/3155887906 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1100f6510 0x7fe110106730 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fe104013510 tx=0x7fe10400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:16.132 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.125+0000 7fe0f77fe700 1 -- 172.21.15.195:0/3155887906 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe104009320 con 0x7fe1100f6510 2023-10-12T22:40:16.132 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.125+0000 7fe0f77fe700 1 -- 172.21.15.195:0/3155887906 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe104013f00 con 0x7fe1100f6510 2023-10-12T22:40:16.132 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.125+0000 7fe115730700 1 -- 172.21.15.195:0/3155887906 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe110106e20 con 0x7fe1100f6510 2023-10-12T22:40:16.133 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.125+0000 7fe0f77fe700 1 -- 172.21.15.195:0/3155887906 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe104014d90 con 0x7fe1100f6510 2023-10-12T22:40:16.133 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.125+0000 7fe115730700 1 -- 172.21.15.195:0/3155887906 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1101072b0 con 0x7fe1100f6510 2023-10-12T22:40:16.133 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.126+0000 7fe0f77fe700 1 -- 172.21.15.195:0/3155887906 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1040095f0 con 0x7fe1100f6510 2023-10-12T22:40:16.134 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.126+0000 7fe115730700 1 -- 172.21.15.195:0/3155887906 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe110045ad0 con 0x7fe1100f6510 2023-10-12T22:40:16.134 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.126+0000 7fe0f77fe700 1 --2- 172.21.15.195:0/3155887906 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe0f802ff60 0x7fe0f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:16.134 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.126+0000 7fe0f77fe700 1 -- 172.21.15.195:0/3155887906 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1040410c0 con 0x7fe1100f6510 2023-10-12T22:40:16.134 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.126+0000 7fe10e7fc700 1 --2- 172.21.15.195:0/3155887906 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe0f802ff60 0x7fe0f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:16.135 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.127+0000 7fe10e7fc700 1 --2- 172.21.15.195:0/3155887906 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe0f802ff60 0x7fe0f8032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fe1000096a0 tx=0x7fe100006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:16.135 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.131+0000 7fe0f77fe700 1 -- 172.21.15.195:0/3155887906 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe10401d7f0 con 0x7fe1100f6510 2023-10-12T22:40:16.342 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.340+0000 7fe115730700 1 -- 172.21.15.195:0/3155887906 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1100fbf60 con 0x7fe1100f6510 2023-10-12T22:40:16.342 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.341+0000 7fe0f77fe700 1 -- 172.21.15.195:0/3155887906 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe104010230 con 0x7fe1100f6510 2023-10-12T22:40:16.343 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:40:16.343 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:40:16.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.342+0000 7fe115730700 1 -- 172.21.15.195:0/3155887906 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe0f802ff60 msgr2=0x7fe0f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:16.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.342+0000 7fe115730700 1 --2- 172.21.15.195:0/3155887906 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe0f802ff60 0x7fe0f8032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fe1000096a0 tx=0x7fe100006b40).stop 2023-10-12T22:40:16.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.342+0000 7fe115730700 1 -- 172.21.15.195:0/3155887906 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1100f6510 msgr2=0x7fe110106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:16.344 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.342+0000 7fe115730700 1 --2- 172.21.15.195:0/3155887906 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1100f6510 0x7fe110106730 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fe104013510 tx=0x7fe10400b040).stop 2023-10-12T22:40:16.345 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.342+0000 7fe115730700 1 -- 172.21.15.195:0/3155887906 shutdown_connections 2023-10-12T22:40:16.345 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.342+0000 7fe115730700 1 --2- 172.21.15.195:0/3155887906 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fe0f802ff60 0x7fe0f8032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:16.345 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.342+0000 7fe115730700 1 --2- 172.21.15.195:0/3155887906 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1100f6510 0x7fe110106730 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:16.345 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.342+0000 7fe115730700 1 -- 172.21.15.195:0/3155887906 >> 172.21.15.195:0/3155887906 conn(0x7fe1100f1c70 msgr2=0x7fe1100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:16.345 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.343+0000 7fe115730700 1 -- 172.21.15.195:0/3155887906 shutdown_connections 2023-10-12T22:40:16.346 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:16.343+0000 7fe115730700 1 -- 172.21.15.195:0/3155887906 wait complete. 2023-10-12T22:40:16.346 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:40:16.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:16 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:40:14.881426+0000 mgr.smithi131.efktma (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:40:17.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:17 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:40:16.342083+0000 mon.smithi131 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.195:0/3155887906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:40:17.740 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:40:17.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:40:18.493 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:18 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:40:16.881868+0000 mgr.smithi131.efktma (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:40:18.974 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.971+0000 7fee367eb700 1 -- 172.21.15.195:0/1109672098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee300f6510 msgr2=0x7fee300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:18.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.971+0000 7fee367eb700 1 --2- 172.21.15.195:0/1109672098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee300f6510 0x7fee300f6930 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fee20009fb0 tx=0x7fee20009b30).stop 2023-10-12T22:40:18.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.971+0000 7fee367eb700 1 -- 172.21.15.195:0/1109672098 shutdown_connections 2023-10-12T22:40:18.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.971+0000 7fee367eb700 1 --2- 172.21.15.195:0/1109672098 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee300f6510 0x7fee300f6930 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:18.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.971+0000 7fee367eb700 1 -- 172.21.15.195:0/1109672098 >> 172.21.15.195:0/1109672098 conn(0x7fee300f1c70 msgr2=0x7fee300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:18.975 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.971+0000 7fee367eb700 1 -- 172.21.15.195:0/1109672098 shutdown_connections 2023-10-12T22:40:18.976 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.971+0000 7fee367eb700 1 -- 172.21.15.195:0/1109672098 wait complete. 2023-10-12T22:40:18.976 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.972+0000 7fee367eb700 1 Processor -- start 2023-10-12T22:40:18.976 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.972+0000 7fee367eb700 1 -- start start 2023-10-12T22:40:18.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.972+0000 7fee367eb700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee300f6510 0x7fee30106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:18.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.972+0000 7fee367eb700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee30106c20 con 0x7fee300f6510 2023-10-12T22:40:18.977 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.972+0000 7fee2ffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee300f6510 0x7fee30106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:18.978 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.972+0000 7fee2ffff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee300f6510 0x7fee30106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51382/0 (socket says 172.21.15.195:51382) 2023-10-12T22:40:18.978 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.972+0000 7fee2ffff700 1 -- 172.21.15.195:0/3498074431 learned_addr learned my addr 172.21.15.195:0/3498074431 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:40:18.978 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.973+0000 7fee2ffff700 1 -- 172.21.15.195:0/3498074431 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee20005040 con 0x7fee300f6510 2023-10-12T22:40:18.979 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.973+0000 7fee2ffff700 1 --2- 172.21.15.195:0/3498074431 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee300f6510 0x7fee30106730 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fee20013510 tx=0x7fee2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:18.979 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.973+0000 7fee2cff9700 1 -- 172.21.15.195:0/3498074431 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee20009320 con 0x7fee300f6510 2023-10-12T22:40:18.979 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.973+0000 7fee367eb700 1 -- 172.21.15.195:0/3498074431 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee30106e20 con 0x7fee300f6510 2023-10-12T22:40:18.980 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.973+0000 7fee2cff9700 1 -- 172.21.15.195:0/3498074431 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee20013f00 con 0x7fee300f6510 2023-10-12T22:40:18.981 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.973+0000 7fee2cff9700 1 -- 172.21.15.195:0/3498074431 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee20014d90 con 0x7fee300f6510 2023-10-12T22:40:18.981 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.973+0000 7fee367eb700 1 -- 172.21.15.195:0/3498074431 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee301072b0 con 0x7fee300f6510 2023-10-12T22:40:18.983 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.974+0000 7fee2cff9700 1 -- 172.21.15.195:0/3498074431 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee20014420 con 0x7fee300f6510 2023-10-12T22:40:18.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.974+0000 7fee367eb700 1 -- 172.21.15.195:0/3498074431 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee30045ad0 con 0x7fee300f6510 2023-10-12T22:40:18.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.974+0000 7fee2cff9700 1 --2- 172.21.15.195:0/3498074431 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fee1802ff60 0x7fee18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:18.984 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.974+0000 7fee2cff9700 1 -- 172.21.15.195:0/3498074431 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee20041080 con 0x7fee300f6510 2023-10-12T22:40:18.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.974+0000 7fee2f7fe700 1 --2- 172.21.15.195:0/3498074431 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fee1802ff60 0x7fee18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:18.985 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.975+0000 7fee2f7fe700 1 --2- 172.21.15.195:0/3498074431 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fee1802ff60 0x7fee18032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fee240096a0 tx=0x7fee24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:18.986 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:18.978+0000 7fee2cff9700 1 -- 172.21.15.195:0/3498074431 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee200095f0 con 0x7fee300f6510 2023-10-12T22:40:19.184 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:19.182+0000 7fee367eb700 1 -- 172.21.15.195:0/3498074431 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee300fbf60 con 0x7fee300f6510 2023-10-12T22:40:19.185 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:19.183+0000 7fee2cff9700 1 -- 172.21.15.195:0/3498074431 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fee2001db70 con 0x7fee300f6510 2023-10-12T22:40:19.185 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:40:19.185 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:40:19.186 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:19.185+0000 7fee367eb700 1 -- 172.21.15.195:0/3498074431 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fee1802ff60 msgr2=0x7fee18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:19.186 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:19.185+0000 7fee367eb700 1 --2- 172.21.15.195:0/3498074431 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fee1802ff60 0x7fee18032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fee240096a0 tx=0x7fee24006b40).stop 2023-10-12T22:40:19.186 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:19.185+0000 7fee367eb700 1 -- 172.21.15.195:0/3498074431 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee300f6510 msgr2=0x7fee30106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:19.187 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:19.185+0000 7fee367eb700 1 --2- 172.21.15.195:0/3498074431 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee300f6510 0x7fee30106730 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fee20013510 tx=0x7fee2000b040).stop 2023-10-12T22:40:19.187 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:19.185+0000 7fee367eb700 1 -- 172.21.15.195:0/3498074431 shutdown_connections 2023-10-12T22:40:19.187 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:19.185+0000 7fee367eb700 1 --2- 172.21.15.195:0/3498074431 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7fee1802ff60 0x7fee18032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:19.187 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:19.185+0000 7fee367eb700 1 --2- 172.21.15.195:0/3498074431 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee300f6510 0x7fee30106730 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:19.187 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:19.185+0000 7fee367eb700 1 -- 172.21.15.195:0/3498074431 >> 172.21.15.195:0/3498074431 conn(0x7fee300f1c70 msgr2=0x7fee300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:19.188 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:19.185+0000 7fee367eb700 1 -- 172.21.15.195:0/3498074431 shutdown_connections 2023-10-12T22:40:19.188 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:19.185+0000 7fee367eb700 1 -- 172.21.15.195:0/3498074431 wait complete. 2023-10-12T22:40:19.188 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:40:19.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:19 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:40:19.184577+0000 mon.smithi131 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.195:0/3498074431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:40:20.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:20 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:40:18.882364+0000 mgr.smithi131.efktma (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:40:20.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-10-12T22:40:20.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-12T22:40:21.733 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.732+0000 7f4bad39a700 1 -- 172.21.15.195:0/4228834492 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ba80f51a0 msgr2=0x7f4ba80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:21.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.732+0000 7f4bad39a700 1 --2- 172.21.15.195:0/4228834492 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ba80f51a0 0x7f4ba80f55c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f4b98009fb0 tx=0x7f4b98009b30).stop 2023-10-12T22:40:21.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.732+0000 7f4bad39a700 1 -- 172.21.15.195:0/4228834492 shutdown_connections 2023-10-12T22:40:21.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.732+0000 7f4bad39a700 1 --2- 172.21.15.195:0/4228834492 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ba80f51a0 0x7f4ba80f55c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:21.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.732+0000 7f4bad39a700 1 -- 172.21.15.195:0/4228834492 >> 172.21.15.195:0/4228834492 conn(0x7f4ba80f09a0 msgr2=0x7f4ba80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:21.734 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.732+0000 7f4bad39a700 1 -- 172.21.15.195:0/4228834492 shutdown_connections 2023-10-12T22:40:21.735 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.732+0000 7f4bad39a700 1 -- 172.21.15.195:0/4228834492 wait complete. 2023-10-12T22:40:21.735 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.732+0000 7f4bad39a700 1 Processor -- start 2023-10-12T22:40:21.735 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.732+0000 7f4bad39a700 1 -- start start 2023-10-12T22:40:21.737 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.733+0000 7f4bad39a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ba80f51a0 0x7f4ba81064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:21.737 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.733+0000 7f4bad39a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba81069e0 con 0x7f4ba80f51a0 2023-10-12T22:40:21.737 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.733+0000 7f4ba7fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ba80f51a0 0x7f4ba81064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:21.737 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.733+0000 7f4ba7fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ba80f51a0 0x7f4ba81064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.195:51392/0 (socket says 172.21.15.195:51392) 2023-10-12T22:40:21.738 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.733+0000 7f4ba7fff700 1 -- 172.21.15.195:0/3897056559 learned_addr learned my addr 172.21.15.195:0/3897056559 (peer_addr_for_me v2:172.21.15.195:0/0) 2023-10-12T22:40:21.738 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.733+0000 7f4ba7fff700 1 -- 172.21.15.195:0/3897056559 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b98005040 con 0x7f4ba80f51a0 2023-10-12T22:40:21.738 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.733+0000 7f4ba7fff700 1 --2- 172.21.15.195:0/3897056559 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ba80f51a0 0x7f4ba81064f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f4b98013510 tx=0x7f4b9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:21.738 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.734+0000 7f4ba4ff9700 1 -- 172.21.15.195:0/3897056559 <== mon.0 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b98009320 con 0x7f4ba80f51a0 2023-10-12T22:40:21.739 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.734+0000 7f4bad39a700 1 -- 172.21.15.195:0/3897056559 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba8106be0 con 0x7f4ba80f51a0 2023-10-12T22:40:21.739 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.734+0000 7f4ba4ff9700 1 -- 172.21.15.195:0/3897056559 <== mon.0 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b98013f00 con 0x7f4ba80f51a0 2023-10-12T22:40:21.739 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.734+0000 7f4ba4ff9700 1 -- 172.21.15.195:0/3897056559 <== mon.0 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b98014dc0 con 0x7f4ba80f51a0 2023-10-12T22:40:21.739 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.734+0000 7f4bad39a700 1 -- 172.21.15.195:0/3897056559 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba8107070 con 0x7f4ba80f51a0 2023-10-12T22:40:21.739 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.734+0000 7f4ba4ff9700 1 -- 172.21.15.195:0/3897056559 <== mon.0 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b98014420 con 0x7f4ba80f51a0 2023-10-12T22:40:21.739 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.734+0000 7f4bad39a700 1 -- 172.21.15.195:0/3897056559 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba8045ad0 con 0x7f4ba80f51a0 2023-10-12T22:40:21.740 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.735+0000 7f4ba4ff9700 1 --2- 172.21.15.195:0/3897056559 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4b9002ff60 0x7f4b90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-10-12T22:40:21.740 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.735+0000 7f4ba4ff9700 1 -- 172.21.15.195:0/3897056559 <== mon.0 v2:172.21.15.131:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b98041110 con 0x7f4ba80f51a0 2023-10-12T22:40:21.740 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.735+0000 7f4ba77fe700 1 --2- 172.21.15.195:0/3897056559 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4b9002ff60 0x7f4b90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-10-12T22:40:21.740 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.735+0000 7f4ba77fe700 1 --2- 172.21.15.195:0/3897056559 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4b9002ff60 0x7f4b90032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f4b9c0096a0 tx=0x7f4b9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-12T22:40:21.741 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.739+0000 7f4ba4ff9700 1 -- 172.21.15.195:0/3897056559 <== mon.0 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b980095f0 con 0x7f4ba80f51a0 2023-10-12T22:40:21.949 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.948+0000 7f4bad39a700 1 -- 172.21.15.195:0/3897056559 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ba80fabe0 con 0x7f4ba80f51a0 2023-10-12T22:40:21.950 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.949+0000 7f4ba4ff9700 1 -- 172.21.15.195:0/3897056559 <== mon.0 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4b9801cb10 con 0x7f4ba80f51a0 2023-10-12T22:40:21.950 INFO:teuthology.orchestra.run.smithi195.stdout: 2023-10-12T22:40:21.950 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"e4069eba-694e-11ee-8db6-212e2dc638e7","modified":"2023-10-12T22:30:29.908459Z","created":"2023-10-12T22:30:29.908459Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-12T22:40:21.951 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.950+0000 7f4bad39a700 1 -- 172.21.15.195:0/3897056559 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4b9002ff60 msgr2=0x7f4b90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:21.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.950+0000 7f4bad39a700 1 --2- 172.21.15.195:0/3897056559 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4b9002ff60 0x7f4b90032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f4b9c0096a0 tx=0x7f4b9c006b40).stop 2023-10-12T22:40:21.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.950+0000 7f4bad39a700 1 -- 172.21.15.195:0/3897056559 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ba80f51a0 msgr2=0x7f4ba81064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-12T22:40:21.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.950+0000 7f4bad39a700 1 --2- 172.21.15.195:0/3897056559 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ba80f51a0 0x7f4ba81064f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f4b98013510 tx=0x7f4b9800b040).stop 2023-10-12T22:40:21.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.950+0000 7f4bad39a700 1 -- 172.21.15.195:0/3897056559 shutdown_connections 2023-10-12T22:40:21.952 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.950+0000 7f4bad39a700 1 --2- 172.21.15.195:0/3897056559 >> [v2:172.21.15.131:6800/3705786429,v1:172.21.15.131:6801/3705786429] conn(0x7f4b9002ff60 0x7f4b90032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:21.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.950+0000 7f4bad39a700 1 --2- 172.21.15.195:0/3897056559 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ba80f51a0 0x7f4ba81064f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-10-12T22:40:21.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.950+0000 7f4bad39a700 1 -- 172.21.15.195:0/3897056559 >> 172.21.15.195:0/3897056559 conn(0x7f4ba80f09a0 msgr2=0x7f4ba805fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-12T22:40:21.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.950+0000 7f4bad39a700 1 -- 172.21.15.195:0/3897056559 shutdown_connections 2023-10-12T22:40:21.953 INFO:teuthology.orchestra.run.smithi195.stderr:2023-10-12T22:40:21.950+0000 7f4bad39a700 1 -- 172.21.15.195:0/3897056559 wait complete. 2023-10-12T22:40:21.954 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2023-10-12T22:40:22.364 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-10-12T22:40:22.365 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-10-12T22:40:22.365 DEBUG:teuthology.orchestra.run.smithi131:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-10-12T22:40:22.389 DEBUG:teuthology.orchestra.run.smithi195:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-10-12T22:40:22.409 INFO:tasks.cephadm:Stopping all daemons... 2023-10-12T22:40:22.410 INFO:tasks.cephadm.mon.smithi131:Stopping mon.smithi131... 2023-10-12T22:40:22.410 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi131 2023-10-12T22:40:22.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: cluster 2023-10-12T22:40:20.882775+0000 mgr.smithi131.efktma (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-12T22:40:22.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: audit 2023-10-12T22:40:21.950099+0000 mon.smithi131 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.195:0/3897056559' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-12T22:40:22.739 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:22 smithi131 systemd[1]: Stopping Ceph mon.smithi131 for e4069eba-694e-11ee-8db6-212e2dc638e7... 2023-10-12T22:40:22.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:40:22.531+0000 7f7ed0d14700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi131 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-10-12T22:40:22.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Oct 12 22:40:22 smithi131 ceph-e4069eba-694e-11ee-8db6-212e2dc638e7-mon.smithi131[112411]: debug 2023-10-12T22:40:22.531+0000 7f7ed0d14700 -1 mon.smithi131@0(leader) e1 *** Got Signal Terminated *** 2023-10-12T22:40:22.861 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi131.service' 2023-10-12T22:40:22.955 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-12T22:40:22.956 INFO:tasks.cephadm.mon.smithi131:Stopped mon.smithi131 2023-10-12T22:40:22.956 INFO:tasks.cephadm.mon.smithi195:Stopping mon.smithi195... 2023-10-12T22:40:22.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi195 2023-10-12T22:40:22.991 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi195.service: Unit ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi195.service not loaded. 2023-10-12T22:40:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-10-12T22:40:22.993 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi195" Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 5: 'sudo systemctl stop ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi195' 2023-10-12T22:40:22.993 INFO:tasks.cephadm:Archiving crash dumps... 2023-10-12T22:40:22.995 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/crash to /home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697/remote/ubuntu@smithi131.front.sepia.ceph.com/crash 2023-10-12T22:40:22.996 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/crash -- . 2023-10-12T22:40:23.052 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/crash to /home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697/remote/ubuntu@smithi195.front.sepia.ceph.com/crash 2023-10-12T22:40:23.054 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/crash -- . 2023-10-12T22:40:23.098 INFO:teuthology.orchestra.run.smithi195.stderr:tar: /var/lib/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/crash: Cannot open: No such file or directory 2023-10-12T22:40:23.098 INFO:teuthology.orchestra.run.smithi195.stderr:tar: Error is not recoverable: exiting now 2023-10-12T22:40:23.100 INFO:tasks.cephadm:Checking cluster log for badness... 2023-10-12T22:40:23.101 DEBUG:teuthology.orchestra.run.smithi131:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-12T22:40:23.131 INFO:teuthology.orchestra.run.smithi131.stderr:grep: /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/ceph.log: No such file or directory 2023-10-12T22:40:23.133 INFO:tasks.cephadm:Compressing logs... 2023-10-12T22:40:23.134 DEBUG:teuthology.orchestra.run.smithi131:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-12T22:40:23.176 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-12T22:40:23.201 INFO:teuthology.orchestra.run.smithi131.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-10-12T22:40:23.201 INFO:teuthology.orchestra.run.smithi131.stderr:: No such file or directory 2023-10-12T22:40:23.202 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/ceph-mgr.smithi131.efktma.log 2023-10-12T22:40:23.202 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/ceph-volume.log 2023-10-12T22:40:23.202 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/ceph-mon.smithi131.log 2023-10-12T22:40:23.203 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/ceph-mgr.smithi131.efktma.log: /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/ceph-volume.log: /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/ceph-mon.smithi131.log: 90.8% -- replaced with /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/ceph-volume.log.gz 2023-10-12T22:40:23.205 INFO:teuthology.orchestra.run.smithi195.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log : No such file or directory 2023-10-12T22:40:23.205 INFO:teuthology.orchestra.run.smithi195.stderr: 2023-10-12T22:40:23.207 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-10-12T22:40:23.207 INFO:teuthology.orchestra.run.smithi131.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-10-12T22:40:23.208 INFO:teuthology.orchestra.run.smithi195.stderr: 2023-10-12T22:40:23.208 INFO:teuthology.orchestra.run.smithi195.stderr:real 0m0.014s 2023-10-12T22:40:23.209 INFO:teuthology.orchestra.run.smithi195.stderr:user 0m0.012s 2023-10-12T22:40:23.209 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m0.009s 2023-10-12T22:40:23.277 INFO:teuthology.orchestra.run.smithi131.stderr:gzip: /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/ceph-mgr.smithi131.efktma.log: file size changed while zipping 2023-10-12T22:40:23.277 INFO:teuthology.orchestra.run.smithi131.stderr: 92.7% -- replaced with /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/ceph-mgr.smithi131.efktma.log.gz 2023-10-12T22:40:23.344 INFO:teuthology.orchestra.run.smithi131.stderr: 90.5% -- replaced with /var/log/ceph/e4069eba-694e-11ee-8db6-212e2dc638e7/ceph-mon.smithi131.log.gz 2023-10-12T22:40:23.346 INFO:teuthology.orchestra.run.smithi131.stderr: 2023-10-12T22:40:23.346 INFO:teuthology.orchestra.run.smithi131.stderr:real 0m0.155s 2023-10-12T22:40:23.347 INFO:teuthology.orchestra.run.smithi131.stderr:user 0m0.227s 2023-10-12T22:40:23.347 INFO:teuthology.orchestra.run.smithi131.stderr:sys 0m0.020s 2023-10-12T22:40:23.347 INFO:tasks.cephadm:Archiving logs... 2023-10-12T22:40:23.348 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/log/ceph to /home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697/remote/ubuntu@smithi131.front.sepia.ceph.com/log 2023-10-12T22:40:23.349 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-12T22:40:23.477 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697/remote/ubuntu@smithi195.front.sepia.ceph.com/log 2023-10-12T22:40:23.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-12T22:40:23.509 INFO:tasks.cephadm:Removing cluster... 2023-10-12T22:40:23.509 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 --force 2023-10-12T22:41:58.834 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e4069eba-694e-11ee-8db6-212e2dc638e7 --force 2023-10-12T22:41:59.147 INFO:tasks.cephadm:Removing cephadm ... 2023-10-12T22:41:59.148 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf /home/ubuntu/cephtest/cephadm 2023-10-12T22:41:59.168 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf /home/ubuntu/cephtest/cephadm 2023-10-12T22:41:59.187 INFO:tasks.cephadm:Teardown complete 2023-10-12T22:41:59.187 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 5: 'sudo systemctl stop ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi195' 2023-10-12T22:41:59.271 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0fa1bae249e9411da330c4f355a3d696 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3da7aaee8cb8051dac57683eeb2f18b9da99d2c9/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 5: 'sudo systemctl stop ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi195' 2023-10-12T22:41:59.276 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-10-12T22:41:59.286 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-12T22:41:59.330 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-12T22:41:59.331 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-12T22:41:59.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-12T22:41:59.425 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-12T22:41:59.437 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-12T22:41:59.437 DEBUG:teuthology.orchestra.run.smithi131:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-12T22:41:59.440 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-12T22:41:59.455 INFO:teuthology.orchestra.run.smithi131.stderr:bash: ntpq: command not found 2023-10-12T22:41:59.458 INFO:teuthology.orchestra.run.smithi195.stderr:bash: ntpq: command not found 2023-10-12T22:41:59.459 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-12T22:41:59.460 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2023-10-12T22:41:59.460 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 21 +176us[ +176us] +/- 99ms 2023-10-12T22:41:59.460 INFO:teuthology.orchestra.run.smithi131.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -515us[ -607us] +/- 49ms 2023-10-12T22:41:59.461 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +785us[ +785us] +/- 90ms 2023-10-12T22:41:59.461 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T22:41:59.463 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-12T22:41:59.463 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2023-10-12T22:41:59.464 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 +152us[ +55us] +/- 99ms 2023-10-12T22:41:59.464 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 -520us[ -618us] +/- 49ms 2023-10-12T22:41:59.465 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +766us[ +766us] +/- 90ms 2023-10-12T22:41:59.465 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-12T22:41:59.466 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-12T22:41:59.478 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-12T22:41:59.479 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-12T22:41:59.524 DEBUG:teuthology.orchestra.run.smithi131:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-12T22:41:59.615 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-12T22:41:59.707 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-12T22:41:59.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T22:41:59.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-12T22:41:59.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T22:41:59.775 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-12T22:41:59.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-12T22:41:59.830 INFO:teuthology.task.internal:Duration was 1383.569980 seconds 2023-10-12T22:41:59.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-12T22:41:59.883 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-12T22:41:59.883 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-12T22:41:59.886 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-12T22:41:59.930 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-12T22:41:59.933 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-12T22:42:00.403 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-12T22:42:00.404 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2023-10-12T22:42:00.404 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-12T22:42:00.429 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2023-10-12T22:42:00.430 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-12T22:42:00.459 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-12T22:42:00.460 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-12T22:42:00.477 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-12T22:42:00.549 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-12T22:42:00.550 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-12T22:42:00.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-12T22:42:00.644 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-12T22:42:00.707 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-12T22:42:00.707 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-12T22:42:00.738 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-12T22:42:00.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-12T22:42:00.864 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-12T22:42:00.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-12T22:42:00.892 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2023-10-12T22:42:00.902 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2023-10-12T22:42:00.936 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-12T22:42:00.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T22:42:00.986 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-12T22:42:01.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-12T22:42:01.004 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-12T22:42:01.014 INFO:teuthology.task.internal:Transferring archived files... 2023-10-12T22:42:01.015 DEBUG:teuthology.misc:Transferring archived files from smithi131:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697/remote/smithi131 2023-10-12T22:42:01.017 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-12T22:42:01.068 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697/remote/smithi195 2023-10-12T22:42:01.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-12T22:42:01.113 INFO:teuthology.task.internal:Removing archive directory... 2023-10-12T22:42:01.114 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-12T22:42:01.116 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-12T22:42:01.162 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-12T22:42:01.181 INFO:teuthology.task.internal:Not uploading archives. 2023-10-12T22:42:01.182 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-12T22:42:01.215 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-12T22:42:01.216 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-12T22:42:01.218 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-12T22:42:01.234 INFO:teuthology.orchestra.run.smithi131.stdout: 393225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 12 22:42 /home/ubuntu/cephtest 2023-10-12T22:42:01.236 INFO:teuthology.orchestra.run.smithi195.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 12 22:42 /home/ubuntu/cephtest 2023-10-12T22:42:01.238 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-12T22:42:01.267 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-12T22:42:01.365 INFO:teuthology.nuke:Checking targets against current locks 2023-10-12T22:42:01.393 DEBUG:teuthology.nuke:shortname: smithi131 2023-10-12T22:42:01.394 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-12T22:42:01.422 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-12 22:10:28.841347', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDFLy6TIRluuOU2mpRGEf+CPBYDhm02Kpe8KbVhgrHGHoHoTGdSrAkVbX8kXk+HL8vZuj+q8217tuUW8g3RFXjk='} 2023-10-12T22:42:01.426 DEBUG:teuthology.nuke:shortname: smithi195 2023-10-12T22:42:01.427 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-12T22:42:01.453 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-10-12_22:09:36-orch:cephadm-wip-adk2-testing-2023-10-11-1434-pacific-distro-default-smithi/7423697', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-12 22:10:28.842732', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKqNVhDHWHENBC2MC/Z8H7Zf/Gxq9arVln7DmZtvPwArFWpMD0GrRioieAvw9z9E7fsHGA5ZVQC1nz/6g3XO/to='} 2023-10-12T22:42:01.482 INFO:teuthology.orchestra.console.smithi131:Power off 2023-10-12T22:42:01.483 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-12T22:42:01.507 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2023-10-12T22:42:01.507 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T22:42:01.525 INFO:teuthology.orchestra.console.smithi195:Power off 2023-10-12T22:42:01.526 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-12T22:42:01.535 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2023-10-12T22:42:01.550 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2023-10-12T22:42:01.550 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T22:42:01.577 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2023-10-12T22:42:05.537 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T22:42:05.579 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T22:42:05.652 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2023-10-12T22:42:05.690 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2023-10-12T22:42:09.654 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T22:42:09.690 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T22:42:09.765 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2023-10-12T22:42:09.802 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2023-10-12T22:42:09.903 INFO:teuthology.orchestra.console.smithi195:Power off completed 2023-10-12T22:42:13.770 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-12T22:42:13.881 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2023-10-12T22:42:13.982 INFO:teuthology.orchestra.console.smithi131:Power off completed 2023-10-12T22:42:14.088 INFO:teuthology.run:Summary data: description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1383.5699796676636 failure_reason: 'Command failed on smithi195 with status 5: ''sudo systemctl stop ceph-e4069eba-694e-11ee-8db6-212e2dc638e7@mon.smithi195''' flavor: default owner: scheduled_adking@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0fa1bae249e9411da330c4f355a3d696 status: fail success: false 2023-10-12T22:42:14.088 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-12T22:42:14.167 INFO:teuthology.run:FAIL